Coverage-guided differential testing of TLS implementations based on syntax mutation.

Yan Pan,Yuefei Zhu,Yubo He,Wei Lin,Licheng Wang

doi:10.1371/journal.pone.0262176

Abstract

Transport layer security (TLS) protocol is the most widely used security protocol in modern network communications. However, protocol vulnerabilities caused by the design of the network protocol or its implementation by programmers emerge one after another. Meanwhile, various versions of TLS protocol implementations exhibit different behavioral characteristics. Researchers are attempting to find the differences in protocol implementations based on differential testing, which is conducive to discovering the vulnerabilities. This paper provides a solution to find the differences more efficiently by targeting the TLS protocol handshake process. The differences of different implementations during the fuzzing process, such as code coverage and response data, are taken to guide the mutation of test cases, and the seeds are mutated based on the TLS protocol syntax. In addition, the definition of duplicate discrepancies is theoretically explored to investigate the root cause of the discrepancies and to reduce the number of duplicate cases that are caused by the same reason. Besides, the necessary conditions for excluding duplicate cases are further analyzed to develop the deduplication strategy. The proposed method is developed based on open-source tools, i.e., NEZHA and TLS-diff. Three types of widely used TLS protocol implementations, i.e., OpenSSL, BoringSSL, and LibreSSL, are taken for experimental testing. The experimental results show that the proposed method can effectively improve the ability to find differences between different implementations. Under the same test scale or the same time, the amount of discrepancies increases by about 20% compared to TLS-diff, indicating the effectiveness of the deduplication strategy.

Highlights

With the rapid development of computer networks, more and more applications are being transformed into network applications
To more effectively detect differences in Transport layer security (TLS) implementation during the handshake process, this paper proposes a coverage-guided differential testing method based on syntax mutation
This paper proposes a coverage-guided differential testing method based on syntax mutation for TLS, which is mainly focused on handling clientHello packets by TLS, and combines the ideas of syntax mutation and guided testing

Summary

Introduction

With the rapid development of computer networks, more and more applications are being transformed into network applications. As carriers of various network transmissions, network protocols occupy an important position in the entire network and play an essential role in ensuring secure communication between network devices. The logical errors in network protocol design and the bugs implemented by programmers in programming lead to the fragility of protocol implementation. As a widely used encryption protocol, TLS protocol is fundamental to encrypted communication. Due to various versions of TLS protocol, different.

Methods

Results

Conclusion