Collecting, Analyzing and Responding to Enterprise Scale DNS Events

Abstract

DNS is an important data source for security for many reasons. If the DNS infrastructure can be brought down, many networking tasks would be impossible to complete. If the integrity of the mapping between domain names and IP addresses is compromised, attackers can redirect users undetectably to IP addresses of their choosing. And malware of many types must in one way or another use the DNS infrastructure as part of their operations. For example, botnets often use fast flux techniques and domain name generation algorithms to rendezvous with command and control servers.Collecting DNS is a significant challenge. In HP, our core internal DNS clusters process approximately 16 billion DNS packets every day. Ideally, we would like to turn each and every one of those packets into an event for our security information and event management (SIEM) system. However, we would have to grow our SIEM, which is one of the largest deployments in the world, by a factor of six to collect this data. Moreover, traditional logging has a substantial performance impact on the DNS infrastructure, and therefore from an operational perspective enabling logging is also impractical. Finally, DNS servers generally do not log the information necessary to detect many security problems.To deal with these problems we collect and filter this traffic using hardware network packet sniffers, which have no impact on the performance of the DNS servers and allows us to collect all of the information we need for security purposes. We model known good traffic, and discard it, keeping only anomalous data.We developed a custom analytics engine, which analyzes this data looking for evidence of botnet infections, blacklist hits, cloud platform abuse, beaconing, data exfiltration, and cache poisoning attempts. The results of these analyses is turned into a set of alerts which are sent to our Security Operations Center (SOC). We've also developed a user interface including various visualizations to help analysts explore the data.The system has been up and running in HP since June 2014. The SOC processes on average about 20 of our alerts per day, with very low false positive rates. We've worked closely with the SOC to make sure the tool is fully integrated into the workflows that the SOC analysts use and meets the needs of the analysts.In this talk, I will describe our experiences developing this tool and the lessons we learned in the process.