A hierarchical deep reinforcement learning model with expert prior knowledge for intelligent penetration testing

Abstract

Penetration testing (PT) is an effective method to assess the security of a network, mainly carried out by experienced human experts, and is widely applied in practice. It is urgent to develop automated tools to alleviate the pressure of talent shortages. Reinforcement learning (RL) is a promising approach to achieving automated PT. However, the high complexity of PT scenarios and the low sample efficiency of RL hinder its applications in practice. Specifically, it faces two dilemmas: (1) vast state and action spaces and (2) highly ineffective exploration.We propose a hierarchical deep reinforcement learning (HDRL) model with expert prior knowledge to overcome the above dilemmas. The HDRL model mitigates the first dilemma. According to the characteristics of PT, we design the model as a hierarchical structure containing two layers of agents, and the agents as a deep neural network to decompose PT tasks and reduce their complexity. Expert prior knowledge mitigates the second dilemma. It is used as rules and knowledge graphs, carries out action constraints according to the rules, and obtains action advice according to knowledge graphs. The two jointly guide the decision-making of agents to reduce invalid exploration.To verify the effectiveness of the proposed method, we design scenarios based on actual network environments. The experimental results show that our model significantly improves the sample efficiency, greatly reduces the learning time of the agents, and shows good performance on large-scale network scenarios, which has the potential to promote the practical application of intelligent PT based on RL.