Detecting Malicious TLS Network Traffic Based on Communication Channel Features

Abstract

For highly camouflaged Command and Control (C&C) communications, especially those using the Transport Security Layer (TLS) protocol, traditional classifiers which only based on statistical features or TLS handshake features gradually fail to detect such behavior. In this context, exploring features of other dimensions to build a more targeted recognition model is one of the ways to alleviate this problem. This paper proposed a new method of detecting malicious TLS traffic by using the communication channel as the detection unit, and a new set of modeling features for the communication channel was designed, including distribution features, the consistency features and statistical features of TLS communication channel. Experiments show that compared with other two types of features, the consistency features contribute most, and combining these three types of features together can train a better classifier which the precision reaches 92.57%. Comparative experiments show that the proposed method is more advantageous when faced with highly camouflaged TLS flows because the proposed method also achieved highest F1 score, and the accuracy is about 2% higher than the classifier based on the TLS handshake features, and 12% higher than the clustering model based on the statistical features of flow.