Reinforcement Learning Powered Station Parameter Optimized Radiation Therapy (SPORT): A Novel Treatment Planning and Beam Delivery Technique

Abstract

Conventional intensity modulated radiation therapy (IMRT) with a typical 5-20 fixed beams often does not provide sufficient angular sampling required for conformal dose shaping, whereas current volumetric modulated arc therapy (VMAT) discretizes the angular space into equally spaced control points without considering the differential need for intensity modulation of different angles, leading to undersampling at some angles while oversampling at some other angles. Our goal is to develop a node or station parameter optimized radiation therapy (SPORT) strategy with simultaneously optimized angular sampling and beam modulation by leveraging state-of-the-art reinforcement learning and the unique capability of modern digital LINACs in dose delivery through programmable nodal points. We developed a SPORT optimization framework, in which, the process of programming control points (or station parameters) was formulated as a stochastic dynamic programming problem, which was solved by a reinforcement learning-based algorithm. On-policy reinforcement learning method, namely, state-action-reward-state-action (SARSA) was integrated with deep convolutional neural network to predict station parameters by utilizing the patient's anatomical structures meanwhile considering the delivery capability of a typical digital LINAC machine. Here, the deep convolutional neural network estimated the state-action value by using the quality of the plan with current station parameters when a next potential station parameter was selected. The state-action value was then updated by SARSA learning. The quality of the plan was quantified by dosimetry constraints. The model was assessed by a retrospective study on a cohort of patients underwent head-and-neck radiation therapy. Dosimetric analysis and delivery efficiency comparisons were used to evaluate the performance of the proposed framework. Our model was used to generate 16 plans unseen in the original training set. All the plans predicted by our model achieved better dose distributions without violating clinical planning constraints. Moreover, instead of using 4 full standard arcs in the original clinically used plans obtained via manual optimization, the predicted plans only used one full standard arc (about 178 control points) plus boost from a few sub-arcs (less than 30 degrees of gantry angles), which significantly improved the efficiency of the beam delivery. We are in the process of integrating the sub-arcs into the full arc by considering the programmable capability of modern LINACs. We demonstrated that a machine learning-based SPORT framework capable of optimizing the spatial sampling and beam modulation simultaneously for modern radiation therapy. The framework not only significantly improves the quality and efficiency of beam delivery, but also has the potential to be incorporated into current clinical workflow to improve the efficiency of dose planning and delivery.