Automated Malware Detection Based on Novel Network Behavioral Signatures

Abstract

In this paper we introduce the second generation of the experimental detection framework of AIPS system which is used for experimentation with detection models and with their combinations. Our research aims mainly on detection of attacks that abuse vulnerabilities of buffer overflow type, but the final goal is to extend detection techniques to cover various types of vulnerabilities. This article describes the concept of detection framework, updated set of network metrics, provides a design of model architecture and shows an experimental results with draft of framework on the set of laboratory simulated attacks. Index Terms—Artificial intelligence, behavioral signatures, metrics, network security, security, security design. 112 metrics divided into five categories according to their nature. These metrics are used to describe properties of detected attack not upon the fingerprint of common signature, but based on its behavior. During the experiments we found several limitations of the original idea and some parts of the architecture were changed. We extended the metric dataset to 169 metrics containing approximately 4000 parameters and changed the categories to reflect the nature of the new dataset. The main goals of this research is (a) to design the architecture of detection framework that will enhance the overall network security level with the ability to learn new behaviors of attacks without intervention of human by using the expert knowledge from Honeypot (or similar) systems; (b) to find the most suitable set of metrics that will successfully describe the behavior of attacks in the network traffic and will significantly higher the detection rate and lower the false positive rate. In this article we introduce the second generation of the experimental detection framework of AIPS system which is used for experimentation with detection models and with their combinations. The fundamental principle of the detection is based on evaluation of metrics set, which describes the behavior of attack. These metrics are formally specified and extraction of them can be generally realized for each data flow. We could interpret the specification of metrics set as formally extended protocol NetFlow (9), which describes more than statistical properties of network