Low latency cyberattack detection in smart grids with deep reinforcement learning

Abstract

This paper focuses on low latency detection of cyberattacks in smart grids with deep reinforcement learning (DRL). The objective of low latency detection is to minimize detection delay while ensuring high detection accuracy. This is different from conventional detection methods that focus mainly on detection accuracy and pay little attention to detection delay. A lower detection delay can reduce recovery time, thus minimizing service interruption or economic losses due to cyberattacks. Since detection delay is the main design metric, the algorithm is developed by using a non-linear dynamic AC system model with an extended Kalman filter (EKF) to capture power grid state transitions in real-time, while many other works in the literature use a simplified linear DC model. The DRL detection algorithm is developed by using a continuous state space deep Q-network (DQN) on the framework of a Markov decision process (MDP). The new DQN design has two main innovations. First, the MDP state is designed as a sliding window of Rao-statistics of the AC dynamic state estimation residues. The proposed state formulation can accurately capture dynamic power state transitions in real-time. Second, a new reward function is designed to allow a flexible trade-off between detection delays and detection accuracy. The delay-accuracy trade-off can be adjusted by tuning a single parameter in the reward function. Simulation results show that the proposed DQN-based DRL detection algorithm can achieve very low detection delays with high detection accuracy.