Assessing generalizability of Deep Reinforcement Learning algorithms for Automated Vulnerability Assessment and Penetration Testing

Abstract

Modern cybersecurity best practices and standards require continuous Vulnerability Assessment (VA) and Penetration Test (PT). These activities are human- and time-expensive. The research community is trying to propose autonomous or semi-autonomous solutions based on Deep Reinforcement Learning (DRL) agents, but current proposals require further investigations. We observe that related literature reports performance tests of the proposed agents against a limited subset of the hosts used to train the models, thus raising questions on their applicability in realistic scenarios. The main contribution of this paper is to fill this gap by investigating the generalization capabilities of existing DRL agents to extend their VAPT operations to hosts that were not used in the training phase. To this purpose, we define a novel VAPT environment through which we devise multiple evaluation scenarios. While evidencing the limited capabilities of shallow RL approaches, we consider three state-of-the-art deep RL agents, namely Deep Q-Network (DQN), Proximal Policy Optimization (PPO), and Advantage Actor–Critic (A2C), and use them as bases for VAPT operations. The results show that the algorithm using A2C DRL agent outperforms the others because it is more adaptable to unknown hosts and converges faster. Our methodology can guide future researchers and practitioners in designing a new generation of semi-autonomous VAPT tools that are suitable for real-world contexts.