Analyzing Learning-based Encrypted Malware Traffic Classification with AutoML

Abstract

Malware traffic classification is an essential pillar of network intrusion detection systems. The explosive growth of traffic encryption makes it infeasible to classify malware traffic with port-based or signature-based approaches. Nowadays, researchers and industrial developers turn to learning-based approaches for encrypted malware traffic classification, mining the statistical patterns of traffic behaviors. However, different machine learning models with different hyper-parameters can be used, and one can hardly explain why a learning approach works or not. To alleviate this problem, this paper conducts encrypted malware traffic classification with the automated machine learning (AutoML) approach which contains 7 representative models in the pipeline and realizes automated hyper-parameter tuning and model assembling. Experimenting on real-world encrypted malware traffic, this paper analyzes the performance of the ensemble model of AutoML and how each model performs in detail to understand its contribution. Moreover, the analysis of AutoML feature selection shows discriminant features on encrypted malware traffic especially TLS metadata related. The concrete experiments and analysis give insight to the following studies on encrypted malware traffic classification.