Segmented sandboxing - A novel approach to Malware polymorphism detection

Abstract

Malware polymorphic and metamorphic obfuscation techniques combined with so-called "sandboxing evasion techniques" continue to erode the effectiveness of both static detection (signature matching), and dynamic detection (sandboxing). Specifically, signature based techniques are overwhelmed by the sheer number of samples generated from a single seminal binary through the use of polymorphic variations (encryption, ISP obfuscation together with ISP emulators, semantically neutral transformations, and so forth). Anti-virus security vendors often report more than 100,000 new Malware signatures a day. In most cases, the preponderance of these variations can be attributed to just a handful of seminal Malware families. In 2011, FireEye reported that over 50% of observed successful Malware infections were attributable to just 13 Malware families (seminals).1 Similarly, sandboxing2, also known as dynamic Malware detection, has suffered from its own set of limitations. Mainly, (1) Malware writers embed in their code the ability to discover virtualized environments by checking for live internet access, or certain system properties inherent to virtualized environments, (2) Wait and seek (aka dormant Malware), a technique where knowing the execution time limitations of sandboxes, the Malware just waits, and (3) evasion techniques based on diverse communication. While the benefits of either dynamic or static approaches for Malware detection look quite tempting from each of their counterpart's perspectives, their weakness are daunting in their own right as well. In this manuscript we attempted to combine the best part of both approaches, while minimizing the disadvantages of either of them. We call this mixed approach "static Malware detection with segmented sandboxing". It was first developed by modeling the problem from a classical automata theory that leads from a formal problem formulation to a practical solution implementation. Preliminary results have shown that this approach is extremely effective in at least two significant ways. First, it sequentially minimizes both false negatives (misses) and false positives (FPs) enabling response resources to be focused on a more complete set of attacks with far less distraction from false alarms. Second, it overcomes many of the known limitations of sandboxing technology.