In-depth fuzz testing of IKE implementations

Abstract

The correctness of security protocols can be proved using formal models. Typically, a security protocol is proven correct at a high level of abstraction, hiding away the implementation details. To execute the protocol on a real system, software engineers take up the task to implement the protocol. Programming is an error-prone activity: despite the fact that a security protocol is proven correct, its implementation may have vulnerabilities, which can be exploited by attackers. In this work we consider the problem of testing security protocol implementations for vulnerabilities, assuming that a formal model of the protocol is given. Finding security vulnerabilities is nontrivial. Even if a system correctly implements the model and behaves as expected when it performs intended tasks, i.e. tasks that are specified in the model, it is difficult to check that the implementation does not exhibit any additional behaviors, i.e. behaviors that are not specified in the model. Behaviors that are unintentionally implemented in the system can be dangerous and may introduce vulnerabilities in the system. When programmers implement the protocol, they need to consider how the system handles unexpected events, e.g. how the system reacts when the disk is full or how to handle malformed input. Such unexpected events are the low-level details that are typically abstracted away in the formal model. We can virtually split the program into two parts: (1) functional part which handles the expected behavior of the system, and (2) error handling part which implements how the program handles unexpected events. The top 4 most dangerous software errors listed at the Common Weakness Enumeration database1 are due to lack of or improper handling of unexpected inputs [6]. Typically, the functional part of the program is well specified in the model and it is less likely that the programmers will implement it incorrectly. We believe that due to lack of specification on how the implementation