A Cooperative Multi-Agent Q-Learning Control Framework for Real-Time Energy Management in Energy Communities

Abstract

Residential and commercial buildings are responsible for 35% of the EU energy-related greenhouse gas (GHG) emissions. Reducing their emissions is crucial for meeting the challenging EU objective of the agenda for becoming a net-zero continent by 2050. The diffusion and integration of distributed renewable energy sources (RESs) and energy storage systems (ESSs), as well as the creation of energy communities (ECs), have proven to be crucial aspects in reducing GHG emissions. In this context, this article proposes a multi-agent AI-based control framework to solve the EC’s energy management problem in the presence of distributed RESs and ESSs as well as considering a shared ESS. The objectives of the proposed control framework are to satisfy the EC members’ load demand to maximize self-consumption and to manage ESSs charging and discharging processes, to enforce cooperative behavior among the EC members by adopting fair and personalized strategies and to maximize EC members’ profits. The proposed control procedure is based on three sequential stages, each solved by a dedicated local RL agent exploiting the Q-Learning algorithm. To reduce the computational complexity of the proposed approach, specifically defined state aggregation criteria were defined to map the RL agents’ continuous state spaces into discrete state spaces of limited dimensions. During the training phase, the EC members’ profiles and the ESSs’ and RESs’ characteristics were randomly changed to allow the RL agents to learn the correct policy to follow in any given scenario. Simulations proved the effectiveness of the proposed approach for different costumers’ load demand profiles and different EC configurations. Indeed, the trained RL agents proved to be able to satisfy the EC members’ load demands to maximize self-consumption, to correctly use the distributed and shared ESSs, to charge them according to respective personalized criteria and to sell the energy surplus, prioritizing sales to the EC. The proposed control framework also proved to be a useful tool for understanding EC performance in different configurations and, thus, for properly dimensioning the EC elements.