Stateful black-box fuzzing for encryption protocols and its application in IPsec

Abstract

Owing to the rapid development of information security technology, the security analysis of encryption protocols has received widespread attention. In this paper, we propose a stateful black-box encryption protocol fuzzing method to analyze the security of real-world black-box encryption protocol programs and devices. This method does not rely on the source code but uses captured packets as input, performs state selection based on a Monte Carlo tree search algorithm, and processes the encryption/decryption conversion of mutant test cases based on the intermediate mapper. It sends test cases and collects responses in interactive communication with the tested program and dynamically optimizes the corpus based on the collected state information. Based on this method, we develop SBEPFuzz and primarily analyze IPsec. We evaluate SBEPFuzz on six widely used IPsec implementations. The experimental results show that SBEPFuzz achieves higher code coverage, and can discover more protocol state sequences and vulnerabilities. Furthermore, we discover four anomalies, including malformed packets triggering service crashes and abnormal interactions leading to plaintext ID leakage, which also reflect the differences in details among different implementations.