A Distance-Based Method for Building an Encrypted Malware Traffic Identification Framework

Abstract

The popularity of encryption method brings a great challenge to malware traffic identification. Traditional classes defined by expert experience are usually classified based on the host behaviors of malware, such as banking malware or ransomware, which are often irrelevant to its communication traffic behaviors. It leads to the fact that the boundaries of traffic feature dataset of different malware classes are fuzzy and make these traditional classes unhelpful for classification based on traffic features. Meanwhile, traditional machine learning-based encrypted malware traffic identification methods, such as using the multi-classification supervised learning model, are inefficient both in model training and detection, and its detection accuracy cannot meet the demand. In this paper, we propose a distance-based method, which utilizes unsupervised learning algorithm Gaussian mixture model (GMM) and ordering points to identify the clustering structure (OPTICS) to calculate the Distance between malwares and make use of the Distance to define the new malware class called FClass. Then, a set of models are trained by XGBoost algorithm to build an identification framework based on the FClass. The performance of the proposed method has been evaluated by comparing it with the other four methods. The results show that the proposed distance-based method is more efficient and accurate.