Radiation Oncology Incident Learning System Research Articles

Implementation of a patient safety training program in radiation oncology residency: A pilot study.

An educational program using Radiation Oncology-Incident Learning System (RO-ILS) was developed to improve safety culture and training for radiation oncology (RO) residents. The program included a pre-training assessment, interactive training, integration of residents into quality assurance meetings, and a post-training assessment over a 3 month rotation. Twelve residents completed the safety training program. Pre-training assessment mean scores (five-point scale) of experience with Incident Learning Systems (ILS), root-cause analysis (RCA), failure-mode and effect analysis (FMEA), safety training, and culture were 2.3, 2.8, 2.0, 4.0, and 4.4, respectively. Post-training assessment showed a significant increase in ILS 4.0 (p<0.001), RCA 3.8 (p=0.008), and FMEA 3.3 (p=0.006) and safety culture (4.8, p=0.043). Additionally, residents were anonymously surveyed ≥ 10 months after graduation to determine the long-term value of the program. The overall assessment from the graduated residents indicates that this education is valued by RO in many institutions. The majority of the residents are either currently utilizing or plan to utilize the information gained in this program in their new institutions. We report a successful implementation of a safety training program in a RO residency with significant improvements in self-reported confidence with the concepts of ILS, RCA, and FMEA and an improved perception of safety culture. This program can be implemented across all residency programs.

Causal analysis of radiotherapy safety incidents based on a hybrid model of HFACS and Bayesian network.

This research investigates the role of human factors of all hierarchical levels in radiotherapy safety incidents and examines their interconnections. Utilizing the human factor analysis and classification system (HFACS) and Bayesian network (BN) methodologies, we created a BN-HFACS model to comprehensively analyze human factors, integrating the hierarchical structure. We examined 81 radiotherapy incidents from the radiation oncology incident learning system (RO-ILS), conducting a qualitative analysis using HFACS. Subsequently, parametric learning was applied to the derived data, and the prior probabilities of human factors were calculated at each BN-HFACS model level. Finally, a sensitivity analysis was conducted to identify the human factors with the greatest influence on unsafe acts. The majority of safety incidents reported on RO-ILS were traced back to the treatment planning phase, with skill errors and habitual violations being the primary unsafe acts causing these incidents. The sensitivity analysis highlighted that the condition of the operators, personnel factors, and environmental factors significantly influenced the occurrence of incidents. Additionally, it underscored the importance of organizational climate and organizational process in triggering unsafe acts. Our findings suggest a strong association between upper-level human factors and unsafe acts among radiotherapy incidents in RO-ILS. To enhance radiation therapy safety and reduce incidents, interventions targeting these key factors are recommended.

Cherenkov and Plan Integration for Real-Time Delivery Verification: The Opportunity for Automated Visualization and Guidance of All Treatments in EBRT

Initiatives in the national radiation oncology incident learning system (RO-ILS) have been exceptionally useful in discovery of factors that lead to incidents and learning of best practices that can help make the national practice of RO safer as a whole. Incident learning systems come from the flight industry, where both visual cues and instrument control are essential parts of implementation, and similarly both human and instrument tools are used in RO. However, numerous RO studies have reported that human factors are a leading cause of incidents discovered and that timelines, such as QA, setup, delivery, and verification are areas where most incidents are found. Tools such as Cherenkov imaging and SGRT can be used to automate many of the riskiest human decisions and/or providing both human vision and instrument-guided oversight in these areas of treatment delivery process. The value of continuous online imaging is reviewed and the two parts have been tested towards complete automation. The conceptual framework of comparing images to the patient treatment plan is outlined with software examples. Development towards a combined SGRT & Cherenkov imaging system that could achieve fully automated incident detection is outlined. Cherenkov imaging has shown direct visualization of many instances of beam delivery to patients that were sub-optimal. These are seen mostly in isolated cases of normal tissue in the beam where it was not expected, such as limbs, chin, contralateral breast or axilla. Incorrect placement of bolus is also readily visualized. Comparisons can be made on a day-to-day basis, but also on a delivery to plan basis if the plan was incorporated into the treatment delivery process. Analysis of the incidents seen indicates that there are automatable metrics of image quality that could have detected them. Overall, if the system detection of variations were fully automated, these could be detected without human intervention. The capabilities for reducing nearly all human error in setup and delivery are available or emerging, and Cherenkov imaging is perhaps one of the most direct ways to capture and computationally analyze the treatment in real time. These tools require further integration with automated analysis and plan integration, but the initial steps are well underway and individual parts are now possible with advances in R&D of the system integration.

Assessing initial plan check efficacy using TG 275 failure modes and incident reporting.

Plan checks are important components of a robust quality assurance (QA) program. Recently, the American Association of Physicists in Medicine (AAPM) published two reports concerning plan and chart checking, Task Group (TG) 275 and Medical Physics Practice Guideline (MPPG) 11.A. The purpose of the current study was to crosswalk initial plan check failure modes revealed in TG 275 against our institutional QA program and local incident reporting data. Ten physicists reviewed 46 high‐risk failure modes reported in Table S1.A.i of the TG 275 report. The committee identified steps in our planning process which sufficiently checked each failure mode. Failure modes that were not covered were noted for follow‐up. A multidisciplinary committee reviewed the narratives of 1599 locally‐reported incidents in our Radiation Oncology Incident Learning System (ROILS) database and categorized each into the high‐risk TG 275 failure modes. We found that over half of the 46 high‐risk failure modes, six of which were top‐ten failure modes, were covered in part by daily contouring peer‐review rounds, upstream of the traditional initial plan check. Five failure modes were not adequately covered, three of which concerned pregnancy, pacemakers, and prior dose. Of the 1599 incidents analyzed, 710 were germane to the initial plan check, 23.4% of which concerned missing pregnancy attestations. Most, however, were caught prior to CT simulation (98.8%). Physics review and initial plan check were the least efficacious checks, with error detection rates of 31.8% and 31.3%, respectively, for some failure modes. Our QA process that includes daily contouring rounds resulted in increased upstream error detection. This work has led to several initiatives in the department, including increased automation and enhancement of several policies and procedures. With TG 275 and MPPG 11.A as a guide, we strongly recommend that departments consider an internal chart checking policy and procedure review.

Incident Learning During the Early COVID-19 Pandemic

Purpose/Objective(s)To assess the impact of the early COVID-19 pandemic on incident learning through evaluation of events reported to RO-ILS: Radiation Oncology Incident Learning System®. The Radiation Oncology Healthcare Advisory Council (RO-HAC) hypothesized that the COVID-19 pandemic would impact the engagement of RO-ILS participants in reporting to the patient safety organization (PSO) and that the characteristics of the reported events would differ from those reported pre-COVID.Materials/MethodsThe RO-ILS database was queried for events reported to the PSO pre-COVID (from March 1, 2019 to July 31, 2019) and during early COVID (March 1, 2020 to July 31, 2020). Events were then segregated into those submitted by the Top 5 reporting practices and those practices in early COVID hotspot states as identified by the Centers of Disease Control and Prevention (WA, CA, AZ, TX, FL, NY, NJ, CT, MA, PA, MD). Descriptive statistics were used to describe trends in reporting and differences in data elements provided by the practice and RO-HAC pre-COVID and during-COVID.ResultsThere was a 16% absolute reduction in event reporting during-COVID (n = 1255) as compared to pre-COVID (n = 1759). Practices located in COVID-hotspots had a 33% absolute reduction in reporting, while those not in hotspots had a 23% reduction. However, initial analysis did not identify drastic change in event classification. Amongst the Top 5 reporting practices, there was a 48% absolute reduction in incident reporting; of note, three of these practices did not report any events to the PSO during-COVID. During-COVID, errors more often occurred and were discovered during treatment planning, regardless of hotspot status. RO-HAC independently rated more events as moderate-critical pre-COVID (43%) than during COVID (33%), whereas practices rated more events as moderate-severe during-COVID (25%) than pre COVID (18%). Despite an expected trend towards more hypofractionated regimens, there was neither an appreciable difference in the types of treatment techniques for all events nor magnitude of dosimetric deviations associated with incidents pre-COVID and during-COVID.ConclusionReporting to RO-ILS declined during the early COVID-19 pandemic, especially in hotspot areas. This suggests that resources and time were diverted away from incident reporting to address other critical needs. Three of the five top reporting practices that ceased reporting during early COVID have since reported events after the analysis timeframe, suggesting the decline may be temporary. RO-HAC overall rated events as higher severity than the practice regardless of the pandemic. However, the drop in perceived severity by RO-HAC pre and during-COVID may be the result of changes in clarity of information provided by the practice. Stability in event classification suggests that practices continued to report a variety of events.

Natural language processing and machine learning to assist radiation oncology incident learning.

PurposeTo develop a Natural Language Processing (NLP) and Machine Learning (ML) pipeline that can be integrated into an Incident Learning System (ILS) to assist radiation oncology incident learning by semi‐automating incident classification. Our goal was to develop ML models that can generate label recommendations, arranged according to their likelihoods, for three data elements in Canadian NSIR‐RT taxonomy.MethodsOver 6000 incident reports were gathered from the Canadian national ILS as well as our local ILS database. Incident descriptions from these reports were processed using various NLP techniques. The processed data with the expert‐generated labels were used to train and evaluate over 500 multi‐output ML algorithms. The top three models were identified and tuned for each of three different taxonomy data elements, namely: (1) process step where the incident occurred, (2) problem type of the incident and (3) the contributing factors of the incident. The best‐performing model after tuning was identified for each data element and tested on unseen data.ResultsThe MultiOutputRegressor extended Linear SVR models performed best on the three data elements. On testing, our models ranked the most appropriate label 1.48 ± 0.03, 1.73 ± 0.05 and 2.66 ± 0.08 for process‐step, problem‐type and contributing factors respectively.ConclusionsWe developed NLP‐ML models that can perform incident classification. These models will be integrated into our ILS to generate a drop‐down menu. This semi‐automated feature has the potential to improve the usability, accuracy and efficiency of our radiation oncology ILS.

An Investigation of Radiation Treatment Learning Opportunities in Relation to the Radiation Oncology Electronic Medical Record: A Single Institution Experience

PurposeA modern radiation oncology electronic medical record (RO-EMR) system represents a sophisticated human-computer interface with the potential to reduce human driven errors and improve patient safety. As the RO-EMR becomes an integral part of clinical processes, it may be advantageous to analyze learning opportunities (LO) based on their relationship with the RO-EMR. This work reviews one institution's documented LO to: (1) study their relationship with the RO-EMR workflow, (2) identify best opportunities to improve RO-EMR workflow design, and (3) identify current RO-EMR workflow challenges. Methods and MaterialsInternal LO reports for an 11-year contiguous period were categorized by their relationship to the RO-EMR. We also identify the specific components of the RO-EMR used or involved in each LO. Additionally, contributing factor categories from the ASTRO/AAPM sponsored Radiation Oncology Incident Learning System's (RO-ILS) nomenclature was used to characterize LO directly linked to the RO-EMR. ResultsA total of 163 LO from the 11-year period were reviewed and analyzed. Most (77.2%) LO involved the RO-EMR in some way. The majority of the LO were the results of human/manual operations. The most common RO-EMR components involved in the studied LO were documentation related to patient setup, treatment session schedule functionality, RO-EMR used as a communication/note-delivery tool, and issues with treatment accessories. Most of the LO had staff lack of attention and policy not followed as 2 of the highest occurring contributing factors. ConclusionsWe found that the majority of LO were related to RO-EMR workflow processes. The high-risk areas were related to manual data entry or manual treatment execution. An evaluation of LO as a function of their relationship with the RO-EMR allowed for opportunities for improvement. In addition to regular radiation oncology quality improvement review and policy update, automated functions in RO-EMR remain highly desirable.

Human factor associations with safety events in radiation therapy.

Background and purposeIncident learning can reveal important opportunities for safety improvement, yet learning from error is challenged by a number of human factors. In this study, incident learning reports have been analyzed with the human factors analysis classification system (HFACS) to uncover predictive patterns of human contributing factors.Materials and methodsSixteen hundred reports from the Safety in Radiation Oncology incident learning system were filtered for inclusion ultimately yielding 141 reports. A radiotherapy‐specific error type was assigned to each event as were all reported human contributing factors. An analysis of associations between human contributing factors and error types was performed.ResultsMultiple associations between human factors were found. Relationships between leadership and risk were demonstrated with supervision failures. Skill‐based errors (those done without much thought while performing familiar tasks) were found to pose a significant safety risk to the treatment planning process. Errors made during quality assurance (QA) activities were associated with decision‐based errors which indicate lacking knowledge or skills.ConclusionAn application of the HFACS to incident learning reports revealed relationships between human contributing factors and radiotherapy errors. Safety improvement efforts should include supervisory influences as they affect risk and error. An association between skill‐based and treatment planning errors showed a need for more mindfulness in this increasingly automated process. An association between decision and QA errors revealed a need for improved education in this area. These and other findings can be used to strategically advance safety.

Cultural Barriers and Enablers for Event Reporting

The purpose of this work was to identify opportunities for increasing event reporting based on a national survey on patient safety culture in a radiation medicine department. The National Hospital Patient Safety Culture Survey developed by the Agency for Healthcare Research and Quality (AHRQ) was disseminated to all staff members of our multi-center radiation medicine department via a web-based tool (Survey Monkey). The overall survey score was the fraction of responses deemed favorable for patient safety as per AHRQ guidelines. Apart from demographic questions, all 42 Likert-scale questions of the survey were mandatory. Completed surveys were amalgamated, and scores computed overall, per question, and per department work staff division. An additional survey question probed respondents on their annual frequency of event reporting with 6 choices ranging from none to over 21. The fraction of responses other than “None” yielded the event reporting rate. Linear regression analysis was performed for the division event reporting rate with each of the 42 questions scored per division. In addition, members of the incident learning system (ILS) analysis team completed an in-house survey to compare the impact of our internal ILS with the national Radiation Oncology Incident-Learning System (RO-ILS) in our department. The overall patient safety culture score was 67.8% (AHRQ average 65%). The department event reporting rate was 49% (> 50th and <75th percentile of all AHRQ respondents). The event reporting rate trends correlated well with survey scores on 4 of the 42 questions (Pearson correlation coefficient > 0.70, p-value <0.05). These were 1) It is just by chance that more serious mistakes don't happen around here (66.4% disagreed); 2) Staff feel like their mistakes are held against them (54% disagreed); 3) My supervisor/manager says a good word when he/she sees a job done according to established patient safety procedures (71.9% agreed); and 4) Mistakes have led to positive changes here (66.4% agreed). In the 2nd survey 82% of the respondents felt that having an ILS improves patient safety culture. 100% felt that even low-severity reports should be filed as fixing such errors may help improve processes in the long run. Neither ILS had a distinct relative advantage towards making our culture safer, but there were slight differences between perceptions of use patterns. Enablers for event reporting include staff perceiving a systematic department effort towards safety, a non-punitive environment for reporting, management appreciation for staff complying with safety guidelines, and impactful departmental responses to reported mistakes. Opposing perceptions are barriers. Having an ILS alongside a culture that supports even low-severity event reporting is perceived to improve reporting and patient safety. The AHRQ survey may be potentially useful for other departments seeking similar insights.

A Blinded, Prospective Study of Error Detection During Physician Chart Rounds in Radiation Oncology

PurposePeer review during physician chart rounds is a major quality assurance and patient safety step in radiation oncology. However, the effectiveness of chart rounds in detecting problematic treatment plans is unknown. We performed a prospective blinded study of error detection at chart rounds to clarify the effectiveness of this quality assurance step. Methods and MaterialsRadiation Oncology Incident Learning System publications were queried for problematic plans approved for treatment that would be detectable at chart rounds. A resident physician, physicist, and dosimetrist collaboratively generated 20 treatment plans with simulated errors identical in nature to those reported to the Radiation Oncology Incident Learning System. These were inserted randomly into weekly chart rounds over 9 weeks, with a median of 2 problematic plans presented per chart rounds (range, 1-4). Data were collected on detection, attendance, length, and number of cases presented at chart rounds. Data were analyzed using descriptive statistics and univariable logistic regression with odds ratios. ResultsThe median length of chart rounds over the study period was 60 minutes (range, 42-79); median number of cases presented per chart rounds was 45 (range, 38-50). The overall detection rate was 55% (11 of 20). Detection rates were higher for cases presented earlier in chart rounds: 75% versus 25% of problematic plans were detected within 30 minutes of start of chart rounds versus after 30 minutes (odds ratio, 0.11; 95% confidence interval, 0.01–0.88; P = .037). Detection rates showed a trend toward increase during the study period but this was not significant: 33% in weeks 1 to 5 and 73% during weeks 6 to 9 (5.3; 95% confidence interval, 0.78–36; P = .08). ConclusionsThe detection of clinically significant problematic plans during chart rounds could be significantly improved. Problematic plans are more frequently detected earlier in chart rounds and inserting such plans into chart rounds may enhance detection; however, larger studies are needed to confirm these findings. A multi-institutional study is planned.

Automatic Incident Triage in Radiation Oncology Incident Learning System.

The Radiotherapy Incident Reporting and Analysis System (RIRAS) receives incident reports from Radiation Oncology facilities across the US Veterans Health Affairs (VHA) enterprise and Virginia Commonwealth University (VCU). In this work, we propose a computational pipeline for analysis of radiation oncology incident reports. Our pipeline uses machine learning (ML) and natural language processing (NLP) based methods to predict the severity of the incidents reported in the RIRAS platform using the textual description of the reported incidents. These incidents in RIRAS are reviewed by a radiation oncology subject matter expert (SME), who initially triages some incidents based on the salient elements in the incident report. To automate the triage process, we used the data from the VHA treatment centers and the VCU radiation oncology department. We used NLP combined with traditional ML algorithms, including support vector machine (SVM) with linear kernel, and compared it against the transfer learning approach with the universal language model fine-tuning (ULMFiT) algorithm. In RIRAS, severities are divided into four categories; A, B, C, and D, with A being the most severe to D being the least. In this work, we built models to predict High (A & B) vs. Low (C & D) severity instead of all the four categories. Models were evaluated with macro-averaged precision, recall, and F1-Score. The Traditional ML machine learning (SVM-linear) approach did well on the VHA dataset with 0.78 F1-Score but performed poorly on the VCU dataset with 0.5 F1-Score. The transfer learning approach did well on both datasets with 0.81 F1-Score on VHA dataset and 0.68 F1-Score on the VCU dataset. Overall, our methods show promise in automating the triage and severity determination process from radiotherapy incident reports.

Radiation Oncology Incident Learning System (RO-ILS): Increasing stakeholder participation for safety and quality improvement.

232 Background: RO-ILS was launched in 2014 and is free, web-based and in more than 500 U.S. radiation facilities. After RO-ILS was implemented at University of Vermont Medical Center (UVMMC), reporting of radiation incidents decreased and participation by radiation staff was limited. To improve incident reporting and participation, RO-ILS was relaunched for all radiation staff members at UVMMC with emphasis on improved system access and education on RO-ILS programmatic goals. Methods: Prior to RO-ILS, safety/quality incidents at UVMMC were submitted by radiation therapists, dosimetrists and physics staff on paper forms and reviewed monthly by the Radiation Quality Committee. After implementation of RO-ILS in 2016, RO-ILS incidents were reviewed by the UVMMC RO-ILS administrators with no formalized staff feedback. Due to decreasing staff submissions, RO-ILS relaunched September 2018 with increased training, scheduled submission review to radiation staff and identification of department champions. Results: Between April 2014 and May 2019, 270 radiation incidents were reported. Prior to RO-ILS, a median 8 incidents were reported per quarter but decreased to 6 per quarter after RO-ILS. After RO-ILS relaunch, median reported incidents increased to 42 per quarter. Radiation “Near Miss” events pre RO-ILS, post RO-ILS and with RO-ILS relaunch were reduced from 78% to 34% to 9%, while “Operational/Process Improvement” submissions increased from 17% pre RO-ILS to 49% post RO-ILS to 81% after relaunch. After RO-ILS relaunch, staff participation expanded to physicians, nursing and administrative staff for the first time, and physician participation increased from 0 to 50%. Conclusions: Following implementation of RO-ILS at UVMMC, radiation incident reporting initially decreased and the proportion of “Near Miss” reports decreased. After relaunch of RO-ILS, there was substantial increase in incident reporting involving all staff. As RO-ILS evolves at UVMMC, there is continued decrease in near misses and greater emphasis on “Process Improvement”. Continued education, reporting and feedback from RO-ILS submissions is recommended to maintain this high staff participation level.

Attending and Resident Physician Engagement of the Radiation Oncology Incident Learning System (RO-ILS)

RO-ILS is designed to improve quality and patient safety in radiation oncology by facilitating shared learning in a non-punitive environment. Higher rates of incident submissions indicate a strong safety culture and support by physicians, who can role model proactive safety culture for team members and trainees. Yet engagement of physicians stands to be improved. We hypothesize that resident versus attending status and select surveyed attitudes predicts RO-ILS utilization. Institutional submissions in RO-ILS between July 1, 2017 and July 1, 2018 were reviewed. In addition, a survey assessing RO-ILS usage, incident reporting to other team members, and attitudes towards submission, was sent to attending and resident physicians in the department. Univariate logistic regression was used to identify if any surveyed attitude’s continuous Likert score predicted direct submission, and if resident versus attending status predicted 1) direct submission, 2) asking another team member to submit incidents (referred submission), or 3) general submission (direct or referred). The multiple testing for attitudes was corrected by Bonferroni method. The association between attitude as a binary categorical variable and direct/general submission was also assessed by chi-squared test. Physicians who self-identified only accounted for 10.7% of all RO-ILS submissions (18/168 total submissions; 5 from residents, 13 from attendings), and these submissions originated from 6 unique physicians (4 attendings, 2 residents) in a department with 13 attendings and 11 residents. The survey was sent to current 12 attendings and 12 residents with a response rate of 83.3% (20/24 physicians); 45% (9/20 respondents) were attendings, 45% (9/20 respondents) reported having ever directly submitted an incident, and 45% (9/20 respondents) reported having ever asked another team member to submit an incident. Of those who referred submissions, 66.7% (6/9 respondents) referred to a therapist or nurse. Avoiding punitive action (OR 0.42, 95% CI 0.14-0.88, p=0.05) and reluctance to submit an incident related to another member of the team (OR 0.51, 95% CI 0.23-0.95, p=0.05) exhibited tendencies for association with submission (Bonferroni threshold p=0.01). Resident status was a borderline significant predictor of less likely to refer submission (OR 0.11, 95% CI 0.01-0.77, p=0.04), and less likely general submission (OR 0.11, 95% CI 0.01-0.73, p=0.03). Attendings are more likely than residents to utilize RO-ILS and ask another team member to submit incidents. The association of RO-ILS use with attitudes of avoiding punitive action and of reluctance to submit incidents related to another team member requires a greater sample size for confirmation.

Eliminating Daily Shifts, Tattoos, and Skin Marks: Streamlining Isocenter Localization With Treatment Plan Embedded Couch Values for External Beam Radiation Therapy

PurposeThe Radiation Oncology Incident Learning System demonstrated that incorrect or omitted patient shifts during treatment are common near-misses or incidents. This single pediatric hospital quality improvement experience evaluated a markless isocenter localization workflow to improve safety and streamline treatment, obviating the need for daily shifts. Methods and MaterialsPatients undergoing radiation therapy were simulated and treated with indexed immobilization devices. User origins were established at simulation based on a limited set of fixed couch-top references. In treatment planning, shifts from the user origin to the planned isocenter were converted to absolute couch parameters and embedded in the setup field parameters. Thus, the first fraction did not require any shifts. Before kilovoltage imaging, setup verification was often supplemented with surface-guided imaging. After image guidance and final couch adjustments, couch parameters could be reacquired and used for subsequent treatments. No skin marks were used. ResultsOver 3 years, approximately 300 patients were treated with over 5000 treatment fractions using this workflow. There were no wrong-site treatment errors. Approximately a dozen near-miss events related to the daily setup process occurred, largely on the first treatment. Root-cause analysis attributed errors to user origin misidentification, couch parameter miscalculation, incorrect immobilization device use, and immobilization device indexed at the wrong indexing position. Skin marks and tattoos were unnecessary. Continuous quality improvement added additional quality assurance checks, resulting in no near-miss incidents or adverse events in the preceding 12 months. ConclusionWe minimized near-miss incidents by using limited simulation user origins, converting user origin-to-isocenter shifts to absolute couch parameters, and enforcing restrictive tolerance tables to limit delivery parameter changes, coupled with surface guidance and quality assurance tools. This technique can be applied across institutions, age ranges, and tumor types and with or without surface guidance. This workflow has removed a common treatment setup error and the need for skin marks.

Reduction of radiation safety incident reporting following implementation of the radiation oncology incident learning system (RO-ILS).

236 Background: The Radiation Oncology Incident Learning System (RO-ILS) was initiated nationally June 2014 and is free, web-based, and currently used in more than 425 U.S. radiation facilities. RO-ILS was implemented at University of Vermont Medical Center (UVMMC) in October 2016 to facilitate safer, higher quality care. This implementation of RO-ILS was reviewed in order to determine whether the conversion to a new reporting system at UVMMC impacted radiation incident reporting at our institution. Methods: Radiation safety reporting at UVMMC included radiation incidents submitted by radiation therapists, dosimetrists and medical physics. Prior to RO-ILS, safety/quality incidents were submitted via a specified reporting form and submissions were reviewed monthly by the Radiation Oncology Quality Committee. After implementation of RO-ILS, radiation safety incidents were entered in RO-ILS and reviewed by the UVMMC RO-ILS administrator. Radiation incidents reported prior to October 2016 were entered into RO-ILS with the initial safety incident date. Results: Between April 2014 and May 2018, 136 radiation safety incidents were reported. There was a median of 7 incidents reported per quarter, decreasing from 8 to 6 per quarter after RO-ILS was implemented. Similarly, the average incidents per quarter was 8 and decreased from 8.9 to 6.7 per quarter after RO-ILS was implemented. Radiation incident types reported prior to RO-ILS were 78% “Near Miss” events but after RO-ILS decreased to 34%, while “Operational/Process Improvement” incidents increased from 17% pre RO-ILS to 49% post RO-ILS. The rate of radiation incidents reported per new patient starting radiation (the most frequent process associated with reported radiation incidents) was 0.59% and showed no significant trends or shifts before or after implementation of RO-ILS. Conclusions: Following implementation of RO-ILS at UVMMC, reported radiation incidents per quarter and the proportion of “Near Miss” events decreased, while the percentage of “Process Improvement” submissions increased. Further training and awareness of RO-ILS is planned with the goal of increasing staff participation and more robust reporting.

ASTRO's Radiation Oncology-Incident Learning System (RO-ILS): A Multi-site, Multi-faculty Radiation Department Culture Immersion into RO-ILS

The Radiation Oncology-Incident Learning System (RO-ILS) is a web based incident reporting system that provides a global mechanism for collaborative learning within radiation oncology. The RO-ILS mission encourages all members to report events as they happen in the department in a non-punitive environment. In September 2017, our multi-site radiation oncology department converted completely to RO-ILS following a decade of usage of an in-house system. The purpose of this work is to share our preliminary experience with RO-ILS, specifically focusing on comparative trends in event reporting, classification and learning. Between October 2017 and January 2018, events submitted were classified into the following categories: Therapeutic radiation incident, other safety incident, near-miss, unsafe condition, operational/process improvement. A single RO-ILS coordinator reviewed and further characterized these events into the following sub-categories for more detailed analysis and follow up. A multi-disciplinary quality management (QM) team met biweekly to investigate and develop follow-up actions which were communicated to the entire staff at monthly meetings at each geographical site to “close the loop” and demonstrate the effectiveness of event reporting. In a 4-month time period, staff reported 325 events. The distribution of events was as follows: Therapeutic radiation incident (53), Other Safety incident (10), Near-miss (19), Unsafe condition (5), and Operational/Process Improvement (238). Due to the default categories being too shallow, events were reclassified into: Communication, Prescription and Directive mismatch, Film Review, Re-simulation, New Plan, On Treatment Visit, Dosimeter, Physics, Equipment Malfunction, Information Technology, Patient on Break, Consent, No physicist on site, Patient Safety, Treatment terminated, Physics weekly chart checks not completed, Pathology, SOARIAN error, Scheduling, and Accident. Event reporting systems serve as an important function in raising awareness and generating a culture of safety. To date, our department has successfully and effectively rolled out RO-ILS and its key components. However, meaningful expansion of default classifications are recommended. RO-ILS allows our team to “track and trend” events, identify root causes, and implement targeted changes to improve patient care. We have provided feedback to the front lines monthly, closing the loop and encouraging further reporting. The frontline staff feels as though their reporting makes a difference, effecting positive changes in the department. As one of the few large multi-site institutions to fully implement RO-ILS, the amount of entries and expanded classifications submitted will provide valuable feedback to the radiation oncology community.

Common error pathways seen in the RO-ILS data that demonstrate opportunities for improving treatment safety

PurposeThe Radiation Oncology Incident Learning System (RO-ILS) receives event reports from facilities across the country. This effort extracted common error pathways seen in the data. These pathways, expressed as fault trees, demonstrate the need for, and opportunities for, preventing these errors and/or limiting their propagation to treatment. Methods and materialsAs of the third quarter of 2016, 2344 event reports had been submitted to RO-ILS and reviewed. A total of 396 of the reports judged highest priority were rereviewed and assigned up to 3 keywords to classify events. Based on patterns among the keyword assignments, the data were further aggregated into pathways leading to 3 general error types: “problematic plan approved for treatment,” “wrong shift instructions given to therapists,” and “wrong shift performed at treatment.” Fault trees were created showing how different errors at different stages in the treatment process combine to flow into these general error types. ResultsA total of 173 of the 396 (44%) events were characterized as belonging to 1 of these 3 general error types. Ninety-nine events were defined as “problematic plan approved for treatment,” 40 as “wrong shift instructions given to therapists,” and 34 as “wrong shift performed at treatment.” Seventy-six of these events (44%) resulted in incorrectly delivered treatment. Event discovery was by therapists (n = 76), physicists (n = 45), physicians (n = 23), dosimetrists (n = 15), or not identified (n = 9); 5 events were found as a result of the patient questioning the staff. For the event type “problematic plan approved for treatment,” 64 of the 99 (65%) events were attributable to physician error: incorrect target or dosing pattern prescribed. ConclusionsData extracted from RO-ILS event reports demonstrate common error pathways in radiation oncology that propagate all the way to treatment. Additional study and coordination of efforts is needed to develop and share best practices to address the sources of these errors and curtail their propagation.

Incident Learning for Stereotactic Radiation Therapy from RO-ILS: Radiation Oncology Incident Learning System

Incident Learning for Stereotactic Radiation Therapy from RO-ILS: Radiation Oncology Incident Learning System

Evaluation of near-miss and adverse events in radiation oncology using a comprehensive causal factor taxonomy

PurposeIncident learning systems (ILSs) are a popular strategy for improving safety in radiation oncology (RO) clinics, but few reports focus on the causes of errors in RO. The goal of this study was to test a causal factor taxonomy developed in 2012 by the American Association of Physicists in Medicine and adopted for use in the RO: Incident Learning System (RO-ILS). Methods and materialsThree hundred event reports were randomly selected from an institutional ILS database and Safety in Radiation Oncology (SAFRON), an international ILS. The reports were split into 3 groups of 100 events each: low-risk institutional, high-risk institutional, and SAFRON. Three raters retrospectively analyzed each event for contributing factors using the American Association of Physicists in Medicine taxonomy. ResultsNo events were described by a single causal factor (median, 7). The causal factor taxonomy was found to be applicable for all events, but 4 causal factors were not described in the taxonomy: linear accelerator failure (n = 3), hardware/equipment failure (n = 2), failure to follow through with a quality improvement intervention (n = 1), and workflow documentation was misleading (n = 1). The most common causal factor categories contributing to events were similar in all event types. The most common specific causal factor to contribute to events was a “slip causing physical error." Poor human factors engineering was the only causal factor found to contribute more frequently to high-risk institutional versus low-risk institutional events. ConclusionsThe taxonomy in the study was found to be applicable for all events and may be useful in root cause analyses and future studies. Communication and human behaviors were the most common errors affecting all types of events. Poor human factors engineering was found to specifically contribute to high-risk more than low-risk institutional events, and may represent a strategy for reducing errors in all types of events.

Quality Improvement Steps From Implementation of Radiation Oncology Incident Learning System (RO-ILS)

Quality Improvement Steps From Implementation of Radiation Oncology Incident Learning System (RO-ILS)