Assessing the Robustness and Performance of Artificial Intelligence Powered Planning Tools in Clinical Settings

Abstract

Artificial intelligence (AI) driven tools have been maturing in automating the radiation treatment planning process. To prepare for clinical deployment of these tools, it is essential to understand their robustness in clinical scenarios. This study works to fill the existing gap between research initiated and clinically ready AI tools by investigating a clinical assessment approach based on phantom design and planning complexity simulation. The hypothesis is that AI tool assessment program provides more clinically relevant and comprehensive evaluations beyond typical model validation studies.A cylindrical digital phantom was designed in the treatment planning system with an axial diameter of 30 cm and length of 18 cm. The phantom contains key structures involved in pancreas SBRT including the PTV25Gy, PTV33Gy, C-loop, stomach, bowel and liver with their base shape and volume representing the average of 100 clinical SBRT patients. Phantom cases were synthesized to mimic real-life anatomical variations and overlaps through displacement, expansion, and rotation of PTVs and OARs. This study involved a total of 32 simulated cases to test a broad range of planning scenarios. A previously developed deep learning based automatic planning tool was assessed in this study. This AI tool is composed of two deep neural networks (NNs) which predict beam dose and fluence maps sequentially. The goal of treatment planning was to deliver 25 Gy to PTV25 and 33 Gy to PTV33 in 5 fractions via simultaneous integral boost (SIB) while limiting luminal OAR max dose to below 29 Gy. The AI-plan's quality was analyzed against the clinical evaluation criteria, which include PTV V100%, luminal OAR max dose using Dmax and D0.03cc. The passing rate of key clinical criteria were collected to quantify overall robustness.For all scenarios, the mean PTV25 V25Gy of the AI plans was 96.7% while mean PTV33 V33 Gy was 82.2%. Large variation (16.3%) in PTV33 V33 Gy was observed due to anatomical variations, i.e., proximity of luminal structures to PTV33. Mean max dose was 28.55, 27.68, and 24.63 Gy for the C-loop, bowel and stomach respectively. Using D0.03cc as a surrogate for max dose, the value was 28.03, 27.12, and 23.84 Gy for the same respective structures. A max dose constraint of 29 Gy was achieved for 81.3% cases for the C-loop and stomach, and 78.1% for the bowel. Using D0.03cc as a surrogate for max dose, the passing rate was 90.6% for the C-loop and 81.3% for both the bowel and stomach.The results showed promising robustness of AI planning tool for pancreas SBRT, providing important evidence of its readiness for clinical implementation. The established approach could guide the robustness testing and other clinical assessments of AI based treatment planning tools in general, which is an important component for safe clinical implementation.