An integrated critic-actor neural network for reinforcement learning with application of DERs control in grid frequency regulation

Abstract

As the electronically-interfaced distributed energy resources (DERs) grow rapidly in power grid, power demand satisfaction and frequency regulation are two main challenges in control area. However, it is difficult to model the analysis of a large-scale grid as well as design a stable and optimal control scheme. With the support of DERs, this paper proposes an actor-critic neural network that integrates a distributed reinforcement learning control scheme to compensate frequency regulation of power grid. The short-term performance and stability is improved by a deterministic learning algorithm that is used to obtain the approximation of desired control output. Meanwhile, a long-term strategic utility function is estimated by the integrated actor-critic neural network. The mapping from system state and control output to the strategic utility function value is identified by neural network, as well as utilized in sub-optimal control learning for further improvement of long-term system performance. Theoretical analysis guarantees the stability. Frequency deviation, tie-line power flow, and long-term cost are coincident with uniform ultimate boundness (UUB). In addition, the upper bound of long-term system cost is also reckoned. The effectiveness and advantages of proposed scheme are illustrated in two case studies. The simulation results indicate that the proposed scheme has better performance under certain condition, compared with some actor-critic network control schemes in frequency regulation of power grid.