Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms on a Building Energy Demand Coordination Task

Abstract

Periods of high demand for electricity can raise electricity prices for building users. Flattening the electricity demand curve reduces can reduce costs and increase resiliency. We formulate this task as a multi-agent reinforcement learning (MA-RL) problem, to be achieved through demand response and coordination of electricity consuming agents, i.e., buildings. Bechmarks for such MA-RL problems do not exist. Here, we contribute an empirical comparison of three classes of MA-RL algorithms: independent learners, centralized critics with decentralized execution, and value factorization learners. We evaluate these algorithms on an energy coordination task in CityLearn, an Open AI Gym environment. We found independent learners with shaped rewards to be competitive with more complex algorithms. Agents with centralized critics aim to learn a rich joint critic, which may complicate the training process and cause scalability issues. Our findings indicate value factorization learners possess the coordination benefits of centralized critics and match independent learners without individualized reward shaping.