ME-Box: A reliable method to detect malicious encrypted traffic

Abstract

Currently, encryption (such as the Transport Layer Security protocol) is used by increasingly more network applications to protect their security and privacy, while it also benefits network attackers who can encrypt their traffic to evade detection. The detection of malicious encrypted traffic is becoming a critical task for cyber security. To accomplish this task, researchers have proposed several enlightening methods, including decryption followed by deep packet inspection (DPI), direct DPI on ciphertext and identification by machine learning algorithms. However, due to privacy violations or performance limitations, the state-of-the-art is far from satisfactory.In this paper, we propose a novel framework and system called ME-Box (Machine learning and Evidence verification) for reliable detection of malicious encrypted traffic. ME-Box has middleboxes deployed in the network and agents installed on the sending hosts. Middleboxes first evaluate the trust degrees of encrypted flows by machine learning methods. If some flows are classified as suspicious, then middleboxes provide evidence of the evaluation results and request the corresponding session-keys from the agents. The agents verify the evidence, and if it is convincing, respond with the correct session-keys. With the session-keys, middleboxes finally decrypt the suspected encrypted flows and perform conventional DPI using intrusion signatures. We implement a prototype system of ME-Box and test it with real malware traffic. The experimental results show that ME-Box requires no modification of current cryptographic protocols and keeps end-users’ privacy well, and its performance is practically deployable.