A comparison of two blending-based ensemble techniques for network anomaly detection in Spark distributed environment

Abstract

In this paper, two blending-based ensemble models, namely, logistic regression-based blending ensemble and SVM-based blending ensemble have been compared in terms of their total training time in a distributed environment and their detection accuracy rates. To handle process of concept drift two clustering algorithms have been compared for their training times in a distributed environment. Tests have been conducted on different machines by varying the number of executor cores to study time latency in a distributed Spark environment. Logistic regression-based blending ensemble with an accuracy of 93% and an accuracy of 98% using SVM-based blending ensemble was achieved. The proposed models have been evaluated using CIDDS dataset.