Distributed Reinforcement Learning for Cyber-Physical System With Multiple Remote State Estimation Under DoS Attacker

Abstract

In this paper, we consider cyber-physical system (CPS) with multiple remote state estimation under denial-of-service (DoS) attack in infinite time-horizon. The sensors monitor the system and send their local state estimate to remote estimators by choosing the local channels in “State 0” or “State 1”. The aim of sensors is to find policies for choosing local channel in a specific state to transmit message to minimize the total estimation error covariance on account of energy-saving in an infinite time-horizon. The DoS attacker aims to achieve the opposite goal by choosing channels to attack or not. The games between sensors and DoS attacker under two different structures of public information are investigated, that is the open-loop case (where sensors and attacker cannot observe others' behaviors) and the closed-loop case (where sensors and attacker can observe the others' behaviors causally). For the open-loop case with assumption that the DoS attacker can get the information from the remote estimators to the sensors, the distributed reinforcement learning algorithms for sensors and attacker based on local information are proposed to find their Nash equilibrium policies, respectively. Further, we consider in closed loop case that the DoS attacker cannot get the information from the remote estimators to the sensors which leads to asymmetric information between the sensors and attacker. To derive Nash equilibrium policies for sensors and attacker, we convert the original game into a belief-based continuous-state stochastic game. The convergence of distributed reinforcement learning method is proved. Some simulations are presented to demonstrate its effectiveness.