Double Deep $Q$ -Learning-Based Distributed Operation of Battery Energy Storage System Considering Uncertainties

Abstract

$Q$ -learning-based operation strategies are being recently applied for optimal operation of energy storage systems, where, a $Q$ -table is used to store $Q$ -values for all possible state-action pairs. However, $Q$ -learning faces challenges when it comes to large state space problems, i.e., continuous state space problems or problems with environment uncertainties. In order to address the limitations of $Q$ -learning, this paper proposes a distributed operation strategy using double deep $Q$ -learning method. It is applied to managing the operation of a community battery energy storage system (CBESS) in a microgrid system. In contrast to $Q$ -learning, the proposed operation strategy is capable of dealing with uncertainties in the system in both grid-connected and islanded modes. This is due to the utilization of a deep neural network as a function approximator to estimate the $Q$ -values. Moreover, the proposed method can mitigate the overestimation that is the major drawback of the standard deep $Q$ -learning. The proposed method trains the model faster by decoupling the selection and evaluation processes. Finally, the performance of the proposed double deep $Q$ -learning-based operation method is evaluated by comparing its results with a centralized approach-based operation.