Multi-Agent Deep Reinforcement Learning for Coordinated Energy Trading and Flexibility Services Provision in Local Electricity Markets

Abstract

Local electricity markets (LEM) have recently attracted great interest as an effective solution to the challenging problem of distributed energy resources’ (DER) management. However, LEM designs combining the market functions of local energy trading and flexibility services (FS) provision to wider system operators have not attracted sufficient attention. In the context of addressing this research gap, this paper firstly provides a new model-based system-centric formulation for the coordination of such a LEM, which provides a theoretical optimality benchmark. Compared to previous formulations, it considers the time-coupling operating characteristics of flexible DERs, and optimizes the two market functions simultaneously. Furthermore, this paper explores for the very first time a model-free prosumer-centric coordination approach for such a LEM, in order to address the practical limitations of model-based system-centric approaches. This is achieved through a new multi-agent deep reinforcement learning method which combines the beneficial properties of the multi-actor-attention-critic and the prioritized experience replay approaches. Case studies on a real-world, large-scale setting validate that the proposed LEM design successfully encapsulates the economic benefits of both local energy trading and FS provision functions, and demonstrate that the proposed learning method outperforms previous methods.