Reinforcement Learning-Based Multi-Phase Seed Scheduling for Network Protocol Fuzzing

Abstract

Listen

Translate article

In network protocol fuzzing, effective seed scheduling plays a critical role in improving testing efficiency. Traditional state-driven seed scheduling methods in network protocol fuzzing are often limited by imbalanced seed selection, monolithic scheduling strategies, and ineffective power allocation. To overcome these limitations, we propose SCFuzz, specifically by employing a multi-armed bandit model to dynamically balance exploration and exploitation across multiple fuzzing phases. The fuzzing process is divided into initial, middle, and final phases with seed selection strategies adapted at each phase to optimize the discovery of new states, paths, and code coverage. Additionally, SCFuzz employs a power allocation method based on state weights, focusing power on high-potential messages to improve the overall fuzzing efficiency. Experimental evaluations on open-source protocol implementations show that SCFuzz significantly improves state and code coverage, achieving up to 17.10% more states, 22.92% higher state transitions, and 7.92% greater code branch coverage compared to AFLNet. Moreover, SCFuzz improves seed selection effectiveness by 389.37% and increases power utilization by 45.61%, effectively boosting the overall efficiency of fuzzing.