Data-driven constrained reinforcement learning for optimal control of a multistage evaporation process

Abstract

It is challenging for reinforcement learning to solve the optimal control problem of industrial processes with constraints under uncertain operating conditions. In this context, this paper proposes a novel data-driven constrained reinforcement learning algorithm for the optimal control of a multistage evaporation process consisting of multiple evaporators in series with coupled liquid levels. We first formulate the optimal control problem as a constrained Markov decision process. Then, with the cumulative tracking error of the outlet liquor density taken as the cumulative constraint, a Lagrangian-based constrained policy optimization is developed. The fast setpoint tracking is achieved by gradient iteration of the policy and the dual variable. An action correction layer based on the online sequential version of random vector functional-link networks is built on the output of the policy network to address the instantaneous constraints of the liquid levels. The infeasible action is corrected in real-time so as to keep the liquid levels in each evaporator within operating range. Finally, we utilize both on-policy and off-policy data generated by the interaction between the constrained policy and the evaporation environment to update our algorithm, which is more data-efficient. Experiments have been carried out on a multistage evaporation system, and the results validate the effectiveness of the proposed algorithm.