Behavioral fine-grained detection and classification of P2P bots

Abstract

Modern botnets are increasingly shifting towards overlay networks, using peer-to-peer (P2P) protocols, for command and control (C&C). P2P botnets are robust against detection and takedown as they avoid single nodes of failure, and mostly use custom encrypted C&C communications. Pattern-based signatures are also inappropriate, yet they cannot efficiently detect malware that uses benign P2P applications such as Kademlia and Overnet. This paper presents PeerMinor, a fully behavioral system that detects and classifies P2P bots inside corporate networks. PeerMinor learns the behavior of known malware and benign P2P applications in order to detect P2P bots and provide security administrators with a correct diagnosis of ongoing malware infections. PeerMinor operates in two phases, learning and detection. In the learning phase, it processes known malware and benign P2P traffic in order to build a two-stage classifier. In the first stage, PeerMinor uses supervised learning in order to build a detection model that separates malicious and benign P2P network activity. In the second stage, it builds a one-class classifier for each known P2P malware family, and uses these classifiers to associate detected P2P bots with a known malware family where possible, thus providing a better situational awareness for system administrators. During detection, PeerMinor processes network traffic using its learning-based model in order to detect P2P bots. To the best of our knowledge, PeerMinor is the first behavioral system that goes beyond simple detection in order to provide an accurate diagnosis about ongoing malware infections. Experimental results prove that PeerMinor achieves both scalability and accuracy. It uses only network features with no need of pattern-based signatures, which can be easily evaded by botnet herders.