An Efficient Parallel Reinforcement Learning Approach to Cross-Layer Defense Mechanism in Industrial Control Systems

Abstract

The ongoing digitalization enables stable control processes and smooth operations of Industrial Control Systems (ICSs). A direct consequence of the highly interconnected architecture of ICSs is the introduced cyber vulnerability and increasing cyber security threats to ICSs. Numerous researches pay attention to the security problem of ICSs. However, most current researches face two challenges. First, the interaction problem between cyber layer and physical layer of ICSs may result incorrect attack response strategies. Second, ICSs are real-time systems, but existing defense decision algorithms based on game theory or reinforcement learning techniques have high computational complexity, which prevents it from making decisions quickly. In this paper, we design a new multi-attribute based reward quantitative method and propose a multi-attribute based Q-learning algorithm to resolve the interaction problem. In addition, to overcome the limitation of slow convergence, we develop an effective parallel Q-learning (PQL) algorithm to quickly find the optimal strategy. The experimental results show the effectiveness of the PQL algorithm. Compared with the Q-learning algorithm (QL) and the deep Q-network (DQN) algorithm, our proposed solution can reduce the average completion time by 12.5%-37%.