DRL-S: Toward safe real-world learning of dynamic thermal management in data center

Abstract

Deep Reinforcement Learning has been researched for Dynamic Thermal Management in Data Centers. An objective of Dynamic Thermal Management is to minimize power consumption while satisfying a set of hard and soft constraints. However, there is a lack of research on the safety issues of applying Deep Reinforcement Learning in Data Centers during real-world learning, limiting its deployment. To this end, this paper proposes a new method named DRL-S, which can solve both hard and soft constraints simultaneously during the learning stage. In addition to optimizing the main objective, Lagrangian-based Constrained Deep Reinforcement Learning and Reward Shaping enforce policies to satisfy soft constraints through extensive online learning. However, the policy typically fails to satisfy the hard constraints due to the random sampling of actions to encourage exploration and the imperfect policy at the initial online learning stage. To ensure hard constraints are satisfied, we further propose to utilize parameterized Shielding, integrating the approximation of the system dynamics and the projection of the action space to predict the safety of candidate actions and provide backup actions when necessary. Results show that the Lagrangian-based method and Reward Shaping can gradually learn policies to reduce soft constraint violations. The former can better balance the relationship between the main objective and violations by updating Lagrangian multipliers. DRL-S can also effectively avoid extreme temperatures without affecting the normal learning process of vanilla algorithms. The asymptotic power consumption is more than 12% lower than the baseline controller.