Optimal Control of Iron-Removal Systems Based on Off-Policy Reinforcement Learning

Abstract

The goethite iron-removal process is an important procedure to remove the iron ions from the zinc hydrometallurgy. However, as a coherent system with complex reaction mechanism, associated uncertainties, and interconnected adjacent reactors, it is difficult for the process to accurately control the ion concentration. Because a large amount of historical data can be obtained during the process, an optimal control algorithm based on off-policy reinforcement learning is proposed in this paper to overcome these difficulties. According to the historical data, the weights of neural network are learned offline, and the optimal control strategy is solved online. Firstly, a bounded function is introduced to define the maximum effect of the coherent system on the subsystem cost function and to extend the cost function of the nominal system, so that the decentralized guaranteed cost control problem can be expressed as the optimal control problem of the nominal system. Then, an approximate iterative control algorithm based on actor-critic structure is proposed. The actor and critic neural networks are used to approximate control strategies and cost functions respectively. To achieve complete off-line, a new neural network is added to the actor-critic structure to approximate a part of the unknown system structure, and the three neural network parameters are optimized by the state transition algorithm. Finally, the strategy update and strategy iteration operations are performed alternately to learn optimal control strategies. The effectiveness and flexibility of the proposed off-policy optimal control method is validated by data from a real industrial goethite iron-removal process.