Shellix: An Efficient Approach for Shellcode Detection

Abstract

Code injection attacks are widely used by attackers to compromise computer systems. Attackers could obtain the control of a victim's computer system by injecting shellcode to the vulnerable program. The existing solutions to detect shellcode can be grouped into two categories: static analysis and dynamic analysis. Static analysis can detect shellcode efficiently, but cannot handle the shellcode that employs obfuscation techniques. Dynamic analysis is able to detect the obfuscated shellcode, however it is still limited to detect the recent virtualization-aware shellcode. In this paper, we present a novel shellcode detection approach without using any virtualization technology. We implement our approach based on the commodity OS kernel which is compatible to the existing system. Our approach is able to detect the shellcode that could be aware of the virtualization environment and reduces the probability of exposing detection environment. Our experimental evaluations show that our system can effectively detect a large set of shellcode instances with good performance.