DUSC-DQN:An Improved Deep Q-Network for Intelligent Penetration Testing Path Design

Abstract

With the development of internet technology, the security risks in the network are increasing. Malicious network attacks will cause huge losses to users, so it is very important to maintain cybersecurity. Penetration testing is an effective method to test the security of computer systems. Traditional penetration testing methods are operated manually by network security technicians, which greatly consumes human resources. And the testing effect is also affected by the ability of network security technicians. In order to save human cost and improve the efficiency of penetration test, we design an intelligent penetration test path design algorithm called DUSC-DQN, which is based on reinforcement learning. DUSC-DQN adopts a Dueling network mechanism and uses the Epsilon Greedy-UCB algorithm to enhance its exploration ability. It improves the activation function in the neural network to enhance its nonlinear expression ability. DUSC-DQN also uses the CQL algorithm to learn the conservative Q function. Through experimental verification, the proposed method can be better used for intelligent penetration test path design. It will improve the efficiency of penetration test and maintain cybersecurity.