Hierarchical Reinforcement Learning for Blockchain-Assisted Software Defined Industrial Energy Market

Abstract

Energy Internet (EI) is developing and booming rapidly with the increase of distributed energy resources, which is beneficial to address the severe condition of industrial energy. However, there are inevitable credit crises and utility optimization challenges in EI that need to be settled. In this article, we propose a blockchain-assisted software defined energy Internet (BSDEI), where a distributed energy market smart contract is designed to ensure transactions executed reliably and participants’ accounts dealt accurately. In order to jointly optimize the utilities of operators, retailers, and industrial prosumers in BSDEI, we formulate the whole trading process as a three-stage Stackelberg game, with the proof of existence and uniqueness for the Stackelberg equilibrium. Then, we design a hierarchical distributed policy gradient algorithm to solve the Stackelberg game under incomplete information. We implement a blockchain-based industrial energy trading system using a middleware platform. The smart contract is deployed on the consortium blockchain, providing website interfaces for participants to operate. Furthermore, we conduct experiments for analyzing economic benefits. Our system prototype demonstrates the feasibility of BSDEI and the algorithm exceeds about 18% in total mean reward than comparing algorithms.