A Hierarchical Deep Reinforcement Learning-Based Community Energy Trading Scheme for a Neighborhood of Smart Households

Abstract

The uncertainties from the distributed energy resources (DERs) and the prosumer’s heterogeneous characteristics bring great challenges to the operation of the community energy market. In this paper, the real-time energy sharing and management in the community market are decomposed into two-layered sub-problems: the household appliance scheduling problem and the internal energy trading pricing problem. A hierarchical deep reinforcement learning (HDRL) based scheme is proposed for the community energy trading with multiple households, containing a two-stage learning process. In the inner stage, a multi-agent deep reinforcement learning (MADRL) based approach is developed to learn the real-time appliances scheduling policy based on the local observations and given internal electricity price in a decentralized way. In the outer stage, a deep reinforcement learning (DRL) based pricing approach is proposed to determine the real-time internal electricity prices based on the participants’ historical net power and external energy supplier’s electricity prices. The scheduling policy and households’ state are not required in the pricing process. Finally, the simulations with real-world datasets demonstrate that the internal energy trading can significantly reduce the prosumers’ daily cost and the control performance of the proposed scheme is superior to the existing studies.