Decentralized learning of energy optimal production policies using PLC-informed reinforcement learning

Abstract

This paper presents a novel approach to distributed optimization in production systems using reinforcement learning (RL) with particular emphasis on energy efficient production. Unlike existing approaches in RL which learn optimal policies from scratch, we speed-up the learning process by considering available control code of the Programmable Logic Controller (PLC) as a baseline for further optimizations. For such PLC-informed RL, we propose Teacher-Student RL to distill the available control code of the individual modules into a neural network which is subsequently optimized using standard RL. The proposed general framework allows to incorporate PLC control code with different level of detail. We implement the approach on a laboratory scale testbed representing different production scenarios ranging from continuous to batch production. We compare the results for different control strategies which show that comparably simple control logic yields considerable improvements of the optimization compared to learning from scratch. The obtained results underline the applicability and potential of the approach in terms of improved production efficiency while considerably reducing the energy consumption of the production schedules.