Intelligent strategic bidding in competitive electricity markets using multi-agent simulation and deep reinforcement learning

Abstract

Aiming at the lack of comprehension of agents in Multi-Agent Simulation (MAS) based on classic Reinforcement Learning algorithms of competitive electricity markets, an intelligent strategic bidding method using Deep Reinforcement Learning (DRL) and MAS is proposed in this paper, which not only can provide more intelligent strategies for market participants to maximize their profits, but can enhance the performance of simulation models dealing with high-dimensional continuous data in electricity markets. Firstly, a theoretical framework of intelligent strategic bidding in competitive electricity markets based on MAS and DRL is proposed, and the process of intelligent bidding in electricity markets based on MAS and DRL is described. Then, three MAS models of intelligent strategic bidding are built based on three classic DRL algorithms, including Deep Q-Network (DQN), Double Deep Q-Network (DDQN), and Asynchronous n-step Q-learning (Async n-step QL), and three algorithms’ convergence speed, computational efficiency, and response sensitivity are compared and analyzed. Finally, a novel Improved Async n-step QL (IAsync n-step QL) algorithm is proposed, the MAS model based on the IAsync n-step QL algorithm for intelligent strategic bidding is established. Simulation results show that the model using the novel DRL algorithm is more profitable and responsive than the classic DRL algorithms.