Abstract
Formal techniques have been devoted to analyzing whether network protocol specifications violate security policies; however, these methods cannot detect vulnerabilities in the implementations of the network protocols themselves. Symbolic execution can be used to analyze the paths of the network protocol implementations, but for stateful network protocols, it is difficult to reach the deep states of the protocol. This paper proposes a novel model-guided approach to detect vulnerabilities in network protocol implementations. Our method first abstracts a finite state machine (FSM) model, then utilizes the model to guide the symbolic execution. This approach achieves high coverage of both the code and the protocol states. The proposed method is implemented and applied to test numerous real-world network protocol implementations. The experimental results indicate that the proposed method is more effective than traditional fuzzing methods such as SPIKE at detecting vulnerabilities in the deep states of network protocol implementations.
Highlights
Network protocol implementations are often prone to vulnerabilities, and formal verification techniques cannot address the problems in the implementations
Traditional fuzzing and symbolic execution methods do not make full use of the protocol state information; they have difficulty reaching the deep states of network protocol implementations
Because protocol states use the input and output packets to interact with their environment, the Mealy machine is more suitable for making protocol inferences
Summary
Network protocol implementations are often prone to vulnerabilities, and formal verification techniques cannot address the problems in the implementations. Fuzz testing and symbolic execution are widely applied to detect vulnerabilities in network protocol implementations. It is difficult to reach the deep states of the stateful network protocols given the complex interactions and state transitions of these methods, because they do not fully exploit the packet interaction and state transition information. We propose a novel approach that uses an FSM model to guide the symbolic execution. We first build a prototype model-guided symbolic execution system to explore the protocol states and detect vulnerabilities in the deep states of the protocol.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
