Simulation of strategic bidding for battery storage and e-mobility in local flexibility markets with multi-agent reinforcement learning

Abstract

This paper proposes a multi-agent reinforcement learning model to simulate the bidding behavior of battery storages and electric vehicles in a bi-level market model, consisting of zonal market and a consecutive local flexibility market. The grid user behavior is computed by a multi-agent deep deterministic policy gradient algorithm. Exemplary results demonstrate the functionality of the implemented model for six agents. The results show, that the model is able to outperform analytical optimization approaches by learning to use strategic bidding to exploit the potentials at the local flexibility market. Further results indicate that the presented model is capable of exploring gaming strategies to maximize the reward. We further prove the necessity of implementing a multi-agent deep reinforcement algorithm by showing that the policy computation fails when a single agent deep reinforcement learning algorithm is used for each agent independently. Using multi-agent deep reinforcement learning, we can address the non-stationarity caused by interaction of agents in the bi-level market structure.