Anomaly Detection Using Anomalous Behavior at Program Environment Through Relative Difference Between Return Addresses

Abstract

Information security is mandatory for the human population in all aspects of electronic gadgets usage. There are different kinds of attacks and anomalies found during the usage of latest applications, where it has the threat of losing the valuable credentials. The security experts have given different solutions to address various levels such as application and programming environment. The machine learning gives maximum solutions for finding anomalies at application level and tremendous outcomes will occur. But in case of programming, the coding exploits are still vulnerable and causes to create abnormal entries through security breaches which forces the program for malfunctioning. So in this paper we are presenting a new kind of anomaly detection to find different sequence of anomalies while running of infected program with help of different process tracing techniques. Here our proposed work uses Linux platform to grapple the anomalies by generating assembly code and tested various possibilities of attacks in program by modeling their original behavior. The virtual space contents such as address entries (return addresses) are helpful in our work to find any kind of anomaly. In this paper we are also improving the performance of anomaly detection by reducing the size of training and test datasets by computing the relative difference between return address entries. Here we have used standard tracing tricks and tools available in Linux platform and the experimental work done on 16 kinds of attacks, artificial datasets generated from normal runs of test programs and Linux commands, finally compared their performance on artificial datasets collected while program normal runs.