Batch process control based on reinforcement learning with segmented prioritized experience replay

Abstract

Batch process is difficult to control accurately due to their complex nonlinear dynamics and unstable operating conditions. The traditional methods such as model predictive control, will seriously affect control performance when process model is inaccurate. In contrast, reinforcement learning (RL) provides an viable alternative by interacting directly with the environment to learn optimal strategy. This paper proposes a batch process controller based on the segmented prioritized experience replay (SPER) soft actor-critic (SAC). SAC combines off-policy updates and maximum entropy RL with an actor-critic formulation, which can obtain a more robust control strategy than other RL methods. To improve the efficiency of the experience replay mechanism in tasks with long episodes and multiple phases, a new method of sampling experience called SPER is designed in SAC. In addition, a novel reward function is set for the SPER-SAC based controller to deal with the sparse reward. Finally, the effectiveness of the SPER-SAC based controller for batch process examples is demonstrated by comparing with the conventional RL-based control methods.