Comparative Evaluation of Different Multi-Agent Reinforcement Learning Mechanisms in Condenser Water System Control

Abstract

Model-free reinforcement learning (RL) techniques are currently drawing attention in the control of heating, ventilation, and air-conditioning (HVAC) systems due to their minor pre-conditions and fast online optimization. The simultaneous optimal control of multiple HVAC appliances is a high-dimensional optimization problem, which single-agent RL schemes can barely handle. Hence, it is necessary to investigate how to address high-dimensional control problems with multiple agents. To realize this, different multi-agent reinforcement learning (MARL) mechanisms are available. This study intends to compare and evaluate three MARL mechanisms: Division, Multiplication, and Interaction. For comparison, quantitative simulations are conducted based on a virtual environment established using measured data of a real condenser water system. The system operation simulation results indicate that (1) Multiplication is not effective for high-dimensional RL-based control problems in HVAC systems due to its low learning speed and high training cost; (2) the performance of Division is close to that of the Interaction mechanism during the initial stage, while Division’s neglect of agent mutual inference limits its performance upper bound; (3) compared to the other two, Interaction is more suitable for multi-equipment HVAC control problems given its performance in both short-term (10% annual energy conservation compared to baseline) and long-term scenarios (over 11% energy conservation).