Automatic Detection of DGA-Enabled Malware Using SDN and Traffic Behavioral Modeling

Abstract

Enterprise networks are under enormous threats from sophisticated cyber-attacks. Various kinds of malware are installed by attackers on compromised hosts, acting as bots that typically use Domain Generation Algorithms (DGAs) to communicate with their Command and Control (C&C) servers. It is computationally expensive to inspect all network packets of every host connected to a large enterprise network in “real-time” at scale with hundreds of gigabits per second data rates. This paper combines Software Defined Networking (SDN) and machine learning to develop an accurate, cost-effective, and scalable system for detecting infected hosts communicating with external C&C servers, subsequent to the resolution of DGA query names. Our solution dynamically selects network flows for diagnosis by trained models in real-time, and relies more on the behavioral traffic profile, rather than packet content. Our first contribution highlights the prevalence and activity pattern of DGA-enabled malware across internal hosts. We draw insights into the behavioral profile of DGA-enabled malware flows when communicating with C&C servers. For our second contribution, we identify malware traffic attributes and train three specialized one-class classifier models using behavioral attributes of malware HTTP, HTTPS and UDP flows. We develop an SDN-based monitoring system to automatically mirror TCP/UDP flows pertinent to DGA queries for diagnosis by the trained models. Finally, we evaluate the efficacy of our approach by testing suspicious traffic flows (selectively recorded by SDN reactive rules), identifying infected hosts, and verifying our detection with an off-the-shelf Intrusion Detection System (IDS) software tool.