Novel Framework for Identifying Anomalies in High Volume of Data using Robust Machine Learning Algorithm

Abstract

Anomaly detection in unlabelled data is very tedious task using unsupervised learning models and also to study the model validation process. Furthermore, it is also getting difficulty to get large-scale labelled data and evaluate the model performance. In real-world problem, in general, for unlabelled data set, evaluation process not only takes time but also increase the project cost. To minimize this issue, here we proposed a novel framework merging with both the unsupervised and supervised learning process. In this framework, the process will generate global label followed by a majority voting ensemble approach. With the availability of global label, the framework latter established a classifier to train and finally predict the appropriate normal or anomaly label of any unlabelled data. This framework has seven stage or seven process and takes very low computing cost. To design this frame-work, we used five unsupervised learning models including Isolation forest (IF), Local Outlier Factor (LOF), Gaussian Mixture, One Class SVM and Auto encoders, in addition to this, we used six supervised learning models like Random Forest, Logistic Regression, CART, Gaussian NB, K-nearest neighbour (KNN) and XG-Boost.All the applied model performance discussed extensively in the report. It is found the proposed framework has able to enhance the accuracy of anomaly detection rate.