Radiation Oncology Incident Learning System Research Articles

The effectiveness of pretreatment physics plan review for detecting errors in radiation therapy.

The pretreatment physics plan review is a standard tool for ensuring treatment quality. Studies have shown that the majority of errors in radiation oncology originate in treatment planning, which underscores the importance of the pretreatment physics plan review. This quality assurance measure is fundamentally important and central to the safety of patients and the quality of care that they receive. However, little is known about its effectiveness. The purpose of this study was to analyze reported incidents to quantify the effectiveness of the pretreatment physics plan review with the goal of improving it. This study analyzed 522 potentially severe or critical near-miss events within an institutional incident learning system collected over a three-year period. Of these 522 events, 356 originated at a workflow point that was prior to the pretreatment physics plan review. The remaining 166 events originated after the pretreatment physics plan review and were not considered in the study. The applicable 356 events were classified into one of the three categories: (1) events detected by the pretreatment physics plan review, (2) events not detected but "potentially detectable" by the physics review, and (3) events "not detectable" by the physics review. Potentially detectable events were further classified by which specific checks performed during the pretreatment physics plan review detected or could have detected the event. For these events, the associated specific check was also evaluated as to the possibility of automating that check given current data structures. For comparison, a similar analysis was carried out on 81 events from the international SAFRON radiation oncology incident learning system. Of the 356 applicable events from the institutional database, 180/356 (51%) were detected or could have been detected by the pretreatment physics plan review. Of these events, 125 actually passed through the physics review; however, only 38% (47/125) were actually detected at the review. Of the 81 events from the SAFRON database, 66/81 (81%) were potentially detectable by the pretreatment physics plan review. From the institutional database, three specific physics checks were particularly effective at detecting events (combined effectiveness of 38%): verifying the isocenter (39/180), verifying DRRs (17/180), and verifying that the plan matched the prescription (12/180). The most effective checks from the SAFRON database were verifying that the plan matched the prescription (13/66) and verifying the field parameters in the record and verify system against those in the plan (23/66). Software-based plan checking systems, if available, would have potential effectiveness of 29% and 64% at detecting events from the institutional and SAFRON databases, respectively. Pretreatment physics plan review is a key safety measure and can detect a high percentage of errors. However, the majority of errors that potentially could have been detected were not detected in this study, indicating the need to improve the pretreatment physics plan review performance. Suggestions for improvement include the automation of specific physics checks performed during the pretreatment physics plan review and the standardization of the review process.

RO-ILS: Radiation Oncology Incident Learning System data trends 2014-2015.

59 Background: Incident learning is one of the most effective ways to improve quality care. To facilitate patient safety improvement at a national level, American Society for Radiation Oncology (ASTRO) and American Association of Physicists in Medicine (AAPM) launched RO-ILS: Radiation Oncology Incident Learning System in June 2014. RO-ILS mission is to facilitate safer and higher quality care through a shared learning environment that is secure and non-punitive. Methods: To ensure the security and protection of data, ASTRO contracted with Clarity PSO, a federally-certified patient safety organization that operates under the auspices of the Patient Safety and Quality Improvement Act of 2005. Radiation oncology practices sign a no-fee contract with Clarity PSO to participate in RO-ILS and then enter safety data into a customized web-based portal. Submitted data are analyzed and interpreted by the Radiation Oncology Healthcare Advisory Council (RO-HAC), a multi-professional team. Practices receive aggregate quarterly reports and institutional reports when substantial data are submitted. Results: During the first year, 61 US practices (123) facilities signed contracts. 42 practices entered 1259 events and 619 of these events (49%) were submitted to the national database. Types of events included: 242 (39%) incidents that reached the patient with or without harm; 206 (33%) near-misses; and 171 (28%) unsafe conditions. RO-HAC identified risk-prone processes including ineffective communication, compressed timelines to start treatment, changes to treatment during the course of therapy and junior practitioners’ errors not remedied by experienced staff. Conclusions: Data suggests that quality assurance processes were effective in catching errors; however, continued work needs to address the origin of these errors and suggest robust solutions. To facilitate improved communication, effective protocols and software enhancements are recommended to alert staff to changes in patients’ management. Policies and procedures on patient hand-offs, emergency cases and oversight of junior staff will help error mitigation. While in its infancy, RO-ILS provides useful data and will serve to improve the quality and safety of radiotherapy.

RO-ILS: Radiation Oncology Incident Learning System: A report from the first year of experience

PurposeIncident learning is a critical tool to improve patient safety. The Patient Safety and Quality Improvement Act of 2005 established essential legal protections to allow for the collection and analysis of medical incidents nationwide. Methods and materialsWorking with a federally listed patient safety organization (PSO), the American Society for Radiation Oncology and the American Association of Physicists in Medicine established RO-ILS: Radiation Oncology Incident Learning System (RO-ILS). This paper provides an overview of the RO-ILS background, development, structure, and workflow, as well as examples of preliminary data and lessons learned. RO-ILS is actively collecting, analyzing, and reporting patient safety events. ResultsAs of February 24, 2015, 46 institutions have signed contracts with Clarity PSO, with 33 contracts pending. Of these, 27 sites have entered 739 patient safety events into local database space, with 358 events (48%) pushed to the national database. ConclusionsTo establish an optimal safety culture, radiation oncology departments should establish formal systems for incident learning that include participation in a nationwide incident learning program such as RO-ILS.

Can emergent treatments result in more severe errors?: An analysis of a large institutional near-miss incident reporting database

PurposeEmergent radiation treatments may be subject to more errors because of the compressed time frame. Few data exist on the magnitude of this problem or how to guide safety improvement interventions. The purpose of this study is to examine patterns of near-miss events in emergent treatments using a large institutional incident reporting system. Methods and materialsEvents in the incident reporting database from February 2012 to October 2013 were reviewed prospectively by a multidisciplinary team to identify emergent treatments. Reports were scored for potential near-miss risk index (NMRI) on a 0 to 4 scale. Workflow steps of where events originated and were detected were analyzed. Events were categorized by use of the causal factor system from the Radiation Oncology Incident Learning System. Mann-Whitney U tests were used to compare mean NMRI score, and Fisher exact tests were performed to compare the proportion of high-risk events between emergent and nonemergent treatments and between emergent treatments on weekdays and weekends or holidays. ResultsOver the study period, approximately 1600 patients were treated, 190 of them emergently. Seventy-one incident reports were submitted for 55 unique patients. Fewer events were reported for emergent treatments than for nonemergent treatments (0.37 events per new treatment vs 0.86; P < .01). Mean risk index for emergent reports was 1.90 versus 1.48 for nonemergent reports (P < .01). Rate of NMRI 4 was 10% for emergent treatments versus 4% for nonemergent treatments (P < .01). Emergent treatments started on a weekend or holiday had a higher proportion of critical near-miss events than emergent treatments started during the week (37% vs 7.9%, P = .034). ConclusionsIn this study, fewer near-miss incidents were reported per treatment course for emergent treatments. This may be attributable to reporting bias. More importantly, when emergent near misses occur, they are of greater severity.

TU‐CD‐BRD‐04: UCLA Experience, with Focus On Developing Metrics and Using RO‐ILS

It has long been standard practice in radiation oncology to report internally when a patient's treatment has not gone as planned and to report events to regulatory agencies when legally required. Most potential errors are caught early and never affect the patient. Quality assurance steps routinely prevent errors from reaching the patient, and these “near misses” are much more frequent than treatment errors. A growing number of radiation oncology facilities have implemented incident learning systems to report and analyze both errors and near misses. Using the term “incident learning” instead of “event reporting” emphasizes the need to use these experiences to change the practice and make future errors less likely and promote an educational, non‐punitive environment.There are challenges in making such a system practical and effective. Speakers from institutions of different sizes and practice environments will share their experiences on how to make such a system work and what benefits their clinics have accrued. Questions that will be addressed include: How to create a system that is easy for front line staff to access How to motivate staff to report How to promote the system as positive and educational and not punitive or demeaning How to organize the team for reviewing and responding to reports How to prioritize which reports to discuss in depth How not to dismiss the rest How to identify underlying causes How to design corrective actions and implement change How to develop useful statistics and analysis tools How to coordinate a departmental system with a larger risk management system How to do this without a dedicated quality manager Some speakers’ experience is with in‐house systems and some will share experience with the AAPM/ASTRO national Radiation Oncology Incident Learning System (RO‐ILS). Reports intended to be of value nationally need to be comprehensible to outsiders; examples of useful reports will be shown.There will be ample time set aside for audience members to contribute to the discussion.Learning Objectives: Learn how to promote the use of an incident learning system in a clinic. Learn how to convert “event reporting” into “incident learning”. See examples of practice changes that have come out of learning systems. Learn how the RO‐ILS system can be used as a primary internal learning system. Learn how to create succinct, meaningful reports useful to outside readers. Gary Ezzell chairs the AAPM committee overseeing RO‐ILS and has received an honorarium from ASTRO for working on the committee reviewing RO‐ILS reports.Derek Brown is a director of TreatSafely.org.Brett Miller has previously received travel expenses and an honorarium from Varian.Phillip Beron has nothing to report.

TU‐CD‐BRD‐00: Incident Learning / RO‐ILS

It has long been standard practice in radiation oncology to report internally when a patient's treatment has not gone as planned and to report events to regulatory agencies when legally required. Most potential errors are caught early and never affect the patient. Quality assurance steps routinely prevent errors from reaching the patient, and these “near misses” are much more frequent than treatment errors. A growing number of radiation oncology facilities have implemented incident learning systems to report and analyze both errors and near misses. Using the term “incident learning” instead of “event reporting” emphasizes the need to use these experiences to change the practice and make future errors less likely and promote an educational, non‐punitive environment.There are challenges in making such a system practical and effective. Speakers from institutions of different sizes and practice environments will share their experiences on how to make such a system work and what benefits their clinics have accrued. Questions that will be addressed include: How to create a system that is easy for front line staff to access How to motivate staff to report How to promote the system as positive and educational and not punitive or demeaning How to organize the team for reviewing and responding to reports How to prioritize which reports to discuss in depth How not to dismiss the rest How to identify underlying causes How to design corrective actions and implement change How to develop useful statistics and analysis tools How to coordinate a departmental system with a larger risk management system How to do this without a dedicated quality manager Some speakers’ experience is with in‐house systems and some will share experience with the AAPM/ASTRO national Radiation Oncology Incident Learning System (RO‐ILS). Reports intended to be of value nationally need to be comprehensible to outsiders; examples of useful reports will be shown.There will be ample time set aside for audience members to contribute to the discussion.Learning Objectives: Learn how to promote the use of an incident learning system in a clinic. Learn how to convert “event reporting” into “incident learning”. See examples of practice changes that have come out of learning systems. Learn how the RO‐ILS system can be used as a primary internal learning system. Learn how to create succinct, meaningful reports useful to outside readers. Gary Ezzell chairs the AAPM committee overseeing RO‐ILS and has received an honorarium from ASTRO for working on the committee reviewing RO‐ILS reports.Derek Brown is a director of TreatSafely.org.Brett Miller has previously received travel expenses and an honorarium from Varian.Phillip Beron has nothing to report.

TU‐CD‐BRD‐02: Henry Ford Hospital System Experience, with Focus On Motivating and Reviewing

It has long been standard practice in radiation oncology to report internally when a patient's treatment has not gone as planned and to report events to regulatory agencies when legally required. Most potential errors are caught early and never affect the patient. Quality assurance steps routinely prevent errors from reaching the patient, and these “near misses” are much more frequent than treatment errors. A growing number of radiation oncology facilities have implemented incident learning systems to report and analyze both errors and near misses. Using the term “incident learning” instead of “event reporting” emphasizes the need to use these experiences to change the practice and make future errors less likely and promote an educational, non‐punitive environment.There are challenges in making such a system practical and effective. Speakers from institutions of different sizes and practice environments will share their experiences on how to make such a system work and what benefits their clinics have accrued. Questions that will be addressed include: How to create a system that is easy for front line staff to access How to motivate staff to report How to promote the system as positive and educational and not punitive or demeaning How to organize the team for reviewing and responding to reports How to prioritize which reports to discuss in depth How not to dismiss the rest How to identify underlying causes How to design corrective actions and implement change How to develop useful statistics and analysis tools How to coordinate a departmental system with a larger risk management system How to do this without a dedicated quality manager Some speakers’ experience is with in‐house systems and some will share experience with the AAPM/ASTRO national Radiation Oncology Incident Learning System (RO‐ILS). Reports intended to be of value nationally need to be comprehensible to outsiders; examples of useful reports will be shown.There will be ample time set aside for audience members to contribute to the discussion.Learning Objectives: Learn how to promote the use of an incident learning system in a clinic. Learn how to convert “event reporting” into “incident learning”. See examples of practice changes that have come out of learning systems. Learn how the RO‐ILS system can be used as a primary internal learning system. Learn how to create succinct, meaningful reports useful to outside readers. Gary Ezzell chairs the AAPM committee overseeing RO‐ILS and has received an honorarium from ASTRO for working on the committee reviewing RO‐ILS reports.Derek Brown is a director of TreatSafely.org.Brett Miller has previously received travel expenses and an honorarium from Varian.Phillip Beron has nothing to report.

TU‐CD‐BRD‐03: UCSD Experience, with Focus On Implementing Change

It has long been standard practice in radiation oncology to report internally when a patient's treatment has not gone as planned and to report events to regulatory agencies when legally required. Most potential errors are caught early and never affect the patient. Quality assurance steps routinely prevent errors from reaching the patient, and these “near misses” are much more frequent than treatment errors. A growing number of radiation oncology facilities have implemented incident learning systems to report and analyze both errors and near misses. Using the term “incident learning” instead of “event reporting” emphasizes the need to use these experiences to change the practice and make future errors less likely and promote an educational, non‐punitive environment.There are challenges in making such a system practical and effective. Speakers from institutions of different sizes and practice environments will share their experiences on how to make such a system work and what benefits their clinics have accrued. Questions that will be addressed include: How to create a system that is easy for front line staff to access How to motivate staff to report How to promote the system as positive and educational and not punitive or demeaning How to organize the team for reviewing and responding to reports How to prioritize which reports to discuss in depth How not to dismiss the rest How to identify underlying causes How to design corrective actions and implement change How to develop useful statistics and analysis tools How to coordinate a departmental system with a larger risk management system How to do this without a dedicated quality manager Some speakers’ experience is with in‐house systems and some will share experience with the AAPM/ASTRO national Radiation Oncology Incident Learning System (RO‐ILS). Reports intended to be of value nationally need to be comprehensible to outsiders; examples of useful reports will be shown.There will be ample time set aside for audience members to contribute to the discussion.Learning Objectives: Learn how to promote the use of an incident learning system in a clinic. Learn how to convert “event reporting” into “incident learning”. See examples of practice changes that have come out of learning systems. Learn how the RO‐ILS system can be used as a primary internal learning system. Learn how to create succinct, meaningful reports useful to outside readers. Gary Ezzell chairs the AAPM committee overseeing RO‐ILS and has received an honorarium from ASTRO for working on the committee reviewing RO‐ILS reports.Derek Brown is a director of TreatSafely.org.Brett Miller has previously received travel expenses and an honorarium from Varian.Phillip Beron has nothing to report.

TU‐CD‐BRD‐01: Making Incident Learning Practical and Useful: Challenges and Previous Experiences

It has long been standard practice in radiation oncology to report internally when a patient's treatment has not gone as planned and to report events to regulatory agencies when legally required. Most potential errors are caught early and never affect the patient. Quality assurance steps routinely prevent errors from reaching the patient, and these “near misses” are much more frequent than treatment errors. A growing number of radiation oncology facilities have implemented incident learning systems to report and analyze both errors and near misses. Using the term “incident learning” instead of “event reporting” emphasizes the need to use these experiences to change the practice and make future errors less likely and promote an educational, non‐punitive environment.There are challenges in making such a system practical and effective. Speakers from institutions of different sizes and practice environments will share their experiences on how to make such a system work and what benefits their clinics have accrued. Questions that will be addressed include: How to create a system that is easy for front line staff to access How to motivate staff to report How to promote the system as positive and educational and not punitive or demeaning How to organize the team for reviewing and responding to reports How to prioritize which reports to discuss in depth How not to dismiss the rest How to identify underlying causes How to design corrective actions and implement change How to develop useful statistics and analysis tools How to coordinate a departmental system with a larger risk management system How to do this without a dedicated quality manager Some speakers’ experience is with in‐house systems and some will share experience with the AAPM/ASTRO national Radiation Oncology Incident Learning System (RO‐ILS). Reports intended to be of value nationally need to be comprehensible to outsiders; examples of useful reports will be shown.There will be ample time set aside for audience members to contribute to the discussion.Learning Objectives: Learn how to promote the use of an incident learning system in a clinic. Learn how to convert “event reporting” into “incident learning”. See examples of practice changes that have come out of learning systems. Learn how the RO‐ILS system can be used as a primary internal learning system. Learn how to create succinct, meaningful reports useful to outside readers. Gary Ezzell chairs the AAPM committee overseeing RO‐ILS and has received an honorarium from ASTRO for working on the committee reviewing RO‐ILS reports.Derek Brown is a director of TreatSafely.org.Brett Miller has previously received travel expenses and an honorarium from Varian.Phillip Beron has nothing to report.

SU‐E‐T‐511: Inter‐Rater Variability in Classification of Incidents in a New Incident Reporting System

PurposeTo determine how consistent the results of different raters are when reviewing the same cases within the Radiation Oncology Incident Learning System (ROILS).MethodsThree second‐year medical physics graduate students filled out incident reports in spreadsheets set up to mimic ROILS. All students studied the same 33 cases and independently entered their assessments, for a total of 99 reviewed cases. The narratives for these cases were obtained from a published International Commission on Radiological Protection (ICRP) report which included shorter narratives selected from the Radiation Oncology Safety Information System (ROSIS) database. Each category of questions was reviewed to see how consistent the results were by utilizing free‐marginal multirater kappa analysis. The percentage of cases where all raters shared full agreement or full disagreement was recorded to show which questions were answered consistently by multiple raters for a given case. The consistency among the raters was analyzed between ICRP and ROSIS cases to see if either group led to more reliable results.ResultsThe categories where all raters agreed 100 percent in their choices were the event type (93.94 percent of cases 0.946 kappa) and the likelihood of the event being harmful to the patient (42.42 percent of cases 0.409 kappa). The categories where all raters disagreed 100 percent in their choices were the dosimetric severity scale (39.39 percent of cases 0.139 kappa) and the potential future toxicity (48.48 percent of cases 0.205 kappa). ROSIS had more cases where all raters disagreed than ICRP (23.06 percent of cases compared to 15.58 percent, respectively).ConclusionDespite reviewing the same cases, the results among the three raters was widespread. ROSIS narratives were shorter than ICRP, which suggests that longer narratives lead to more consistent results. This study shows that the incident reporting system can be optimized to yield more consistent results.

MO‐DE‐BRA‐02: From Teaching to Learning: Systems‐Based‐Practice and Practice‐Based‐Learning Innovations in Medical Physics Education Programs

Purpose:The increasing complexity in the field of radiation medicine and concomitant rise in patient safety concerns call for enhanced systems‐level training for future medical physicists and thus commensurate innovations in existing educational program curricula. In this work we report on the introduction of three learning opportunities to augment medical physics educational programs towards building systems‐based practice and practice‐based learning competencies.Methods:All initiatives were introduced for senior ‐level graduate students and physics residents in an institution with a newly established medical‐physics graduate program and therapeutic‐physics residency program. The first, centered on incident learning, was based on a spreadsheet tool that incorporated the reporting structure of the Radiation Oncology‐incident Learning System (ROILS), included 120 narratives of published incidents and enabled inter‐rater variability calculations. The second, centered on best‐practices, was a zero‐credit seminar course, where students summarized select presentations from the AAPM virtual library on a weekly basis and moderated class discussions using a point/counterpoint approach. Presentation styles were critiqued. The third; centered on learning‐by‐teaching, required physics residents to regularly explain fundamental concepts in radiological physics from standard textbooks to board certified physics faculty members.Results:Use of the incident‐learning system spreadsheet provided a platform to recast known accidents into the framework of ROILS, thereby increasing awareness of factors contributing to unsafe practice and appreciation for inter‐rater variability. The seminar course enhanced awareness of best practices, the effectiveness of presentation styles and encouraged critical thinking. The learn‐by‐teaching rotation allowed residents to stay abreast of and deepen their knowledge of relevant subjects.Conclusion:The incorporation of systems‐driven initiatives broadens comprehension of the wider systems context of medical physics, enhances awareness of resources for innovation, communication and sustained learning while maintaining a metric‐driven focus on patient safety within the formative phase of student careers. The initiatives were well‐received, feasible, and utilized available or shared‐resources translatable across educational programs.

Patient safety across disciplines: radiation oncology incident learning system.

The Radiation Oncology Incident Learning System is a national error reporting system that is voluntary and confidential. Participants collect and analyze their data, then institute corrective actions so that true incident learning takes place.

ASTRO &amp; AAPM Launch Radiation Oncology Incident Learning System to Improve Patient Safety &amp; Quality

ASTRO &amp; AAPM Launch Radiation Oncology Incident Learning System to Improve Patient Safety &amp; Quality

TH‐E‐19A‐01: Quality and Safety in Radiation Therapy

Clinical radiotherapy data clearly demonstrate the link between the quality and safety of radiation treatments and the outcome for patients. The medical physicist plays an essential role in this process. To ensure the highest quality treatments, the medical physicist must understand and employ modern quality improvement techniques. This extends well beyond the duties traditionally associated with prescriptive QA measures. This session will review the current best practices for improving quality and safety in radiation therapy.General elements of quality management will be reviewed including: what makes a good quality management structure, the use of prospective risk analysis such as FMEA, and the use of incident learning. All of these practices are recommended in society‐level documents and are incorporated into the new Practice Accreditation program developed by ASTRO. To be effective, however, these techniques must be practical in a resource‐limited environment. This session will therefore focus on practical tools such as the newly‐released radiation oncology incident learning system, RO‐ILS, supported by AAPM and ASTRO.With these general constructs in mind, a case study will be presented of quality management in an SBRT service. An example FMEA risk assessment will be presented along with incident learning examples including root cause analysis.As the physicist's role as “quality officer” continues to evolve it will be essential to understand and employ the most effective techniques for quality improvement. This session will provide a concrete overview of the fundamentals in quality and safety.Learning Objectives: Recognize the essential elements of a good quality management system in radiotherapy. Understand the value of incident learning and the AAPM/ASTRO ROILS incident learning system. Appreciate failure mode and effects analysis as a risk assessment tool and its use in resource‐limited environments. Understand the fundamental principles of good error proofing that extends beyond traditional prescriptive QA measures.

WE‐F‐WAB‐01: Safety Improvement Through Incident Learning

Incident learning is a key tool for improving the quality and safety of procedures. Its use has been linked to better patient‐safety outcomes. This session is designed to provide practical advice to the clinical medical physicist on how to run a successful incident learning system in radiation oncology. The session will draw on engaging examples from clinical practice and will focus on the relevance to smaller clinics. The session will include a presentation from a leading physician on clinical perspectives on quality and safety improvement. The session will also introduce the new national incident reporting system being developed by AAPM and ASTRO. This program employs the Patient Safety Organization (PSO) structure. The concept and structure of the PSO system will be discussed by an expert in the field.Learning Objectives:1. Appreciate the value of incident learning for improving patient safety2. Understand the difference between incident learning at the institutional level vs. the national level, and recognize the issues unique to each3. Learn about the new national radiation oncology incident learning system developed by AAPM and ASTRO