Reinforcement learning for fluctuation reduction of wind power with energy storage

Abstract

The intrinsic randomness of renewable energy has a negative impact on the safety of power grid. In this paper, we aim at decreasing large fluctuations of the power output from a wind farm integrated with a battery energy storage system (BESS), so as to improve the stability and quality of the power system. The control method is to dynamically charge or discharge the BESS, coordinated with limited wind curtailment. The fluctuation of total power output is measured by variance, which reflects the risk to the safety of grid. The difficulty is that this dynamic optimization problem does not meet the requirement of a standard Markov decision process (MDP) model, since the variance metric is not additive. To solve this problem, we first propose the sensitivity-based optimization method and derive a difference formula to quantify the variance metrics of the system. Then we implement the optimization approach as reinforcement learning algorithms, in a mode of data-driven. We develop the Q-learning algorithm so that it can be executed online and generate improved policies repeatedly with observed data. Furthermore, we implement Deep Q-Networks (DQN) to handle the difficulty of continuous states. The performance of the proposed algorithms is verified with real data, which demonstrates the effectiveness of our algorithms.