Technique for preparation of large data for machine learning algorithms to generate intrusion detection system

Abstract

Machine Learning has become the norm for creating models that predict and detect security breaches in computer systems. Modelling of machine learning systems requires a huge amount of data for training and testing the machine learning algorithm. The present research is concerned with preparing the data prior to feeding it to machine learning algorithms. The necessity for such preparation occurs due to the physical limitations of spreadsheet applications (with regard to the number of records they can handle). This research describes a simple shell scripting approach to combine, extract, and weed out unwanted records and finally prepare the data for feeding to machine learning algorithms.