TLS fingerprint for encrypted malicious traffic detection with attributed graph kernel

Abstract

Recently, more and more applications have adopted security protocols like Transport Layer Security (TLS) for data encryption. However, these privacy-enhancing approaches have also been abused by the attackers to deliver malicious payloads. Many existing encrypted network classification methods suffer from the imbalanced volume of normal and malicious traffic, which leads to bad model robustness. In this paper, we propose a novel TLS fingerprinting approach to capture the characteristics of encrypted network traffic. The fingerprints are attributed graphs obtained from TLS sessions, which can simultaneously take into consideration the sequential and statistical features of these sessions. As the communication patterns of different applications differ considerably, the graphs representing TLS connections could be used to characterize the network with the help of the graph kernel method, which results in a model with high accuracy in malicious TLS session detection and application discrimination. Moreover, we adopt Locality-Sensitive Hashing (LSH) and filtering techniques to reduce the time cost of our model. Model evaluation on real-world datasets shows that our model is more robust than existing methods presented in this work when the malicious traffic takes up an extremely small portion of the whole traffic.