PenGym: Realistic training environment for reinforcement learning pentesting agents

Abstract

Penetration testing, or pentesting, refers to assessing network system security by trying to identify and exploit any existing vulnerabilities. Reinforcement Learning (RL) has recently become an effective method for creating autonomous pentesting agents. However, RL agents are typically trained in a simulated network environment. This can be challenging when deploying them in a real network infrastructure due to the lack of realism of the simulation-trained agents.In this paper, we present PenGym, a framework for training pentesting RL agents in realistic network environments. The most significant features of PenGym are its support for real pentesting actions, full automation of the network environment creation, and good execution performance. The results of our experiments demonstrated the advantages and effectiveness of using PenGym as a realistic training environment in comparison with a simulation approach (NASim). For the largest scenario, agents trained in the original NASim environment behaved poorly when tested in a real environment, having a high failure rate. In contrast, agents trained in PenGym successfully reached the pentesting goal in all our trials. Even after fixing logical modeling issues in simulation to create the revised version NASim(rev.), experiment results with the largest scenario indicated that agents trained in PenGym slightly outperformed, and were more stable, than those trained in NASim(rev.). Thus, the average number of steps required to reach the pentesting goal was 1.4 to 8 steps better for PenGym. Consequently, PenGym provides a reliable and realistic training environment for pentesting RL agents, eliminating the need to model agent actions via simulation.