Network Anomaly Detection Systems Research Articles

Robust network anomaly detection using ensemble learning approach and explainable artificial intelligence (XAI)

Intrusion Detection Systems, specifically Network Anomaly Detection Systems (NADSs) are vital tools in network security. The NADSs are affected by data imbalance issues in classifying minority classes. Also, designing an efficient detection framework is sought after to achieve a higher detection rate for minority classes, especially when utilizing ensemble learning methods. To solve the issue of imbalanced data, a hybrid method of sampling techniques is proposed. This imbalance processing tool integrates the Synthetic Minority Oversampling Technique (SMOTE) and the K-means clustering algorithm (SKM). SMOTE over-samples the minority class, and K-means is used to perform a cluster-based under-sampling. We use Denoising Autoencoder (DAE) to select the top 15 features to reduce data dimensionality based on their higher weights. For anomaly detection, the XGBoost algorithm is deployed and the SHapley Additive exPlanation (SHAP) approach is deployed to provide explanations of the proposed techniques. The performance of the SKM-XGB model is assessed using the NSL-KDD and UNSW-NB15 datasets. A comparative analysis and series of experiments were carried out using several ensemble models with multiple base classifiers. The experimental findings indicate that the model's detection rate for binary classification and multiclass classification using the UNSW-NB15 dataset is 99.01% and 97.49%, respectively. The model achieves a 99.37% detection rate for binary classification and a 99.22% detection rate for multiclass classification on the NSL-KDD dataset. We conducted a comparative analysis of various ensemble models with multiple base classifiers. The results indicate that SKM-XGB outperforms the other investigated models and outperforms the performance of state-of-the-art models.

CRND: An Unsupervised Learning Method to Detect Network Anomaly

Network anomaly detection system (NADS) is one of the most important methods to maintain network system security. At present, network anomaly detection models based on deep learning have become a research hotspot in the area because of their advantage in processing high-dimensional data and excellent performance on detecting anomaly. However, most of the related research studies are based on supervised learning, which has strict requirements for dataset such as labels with high accuracy. However, there are some difficulties in obtaining a large amount of data with complete label message, thus seriously hindering the development and deployment of NADS based on DL. In this paper, we propose an unsupervised learning method to detect network anomaly, contrastive representation for network data (CRND). Based on contrastive learning, without label message, a qualified model is trained, providing more possibilities for the field. On CICIDS2018, the evaluation experiment proves that CRND can achieve 96.13% accuracy with only 200 items, and its F1-score reaches 0.96, which is far higher than that of other existing unsupervised learning methods. As fine-tuning is carried out, F1-score can reach a convergence level of 0.99, and the detection performance is the same as that of the detection model based on supervised learning.

Network Intrusion Detection with 1D Convolutional Neural Networks

Computer network assets expose to various cyber threats in today’s digital era. Network Anomaly Detection Systems (NADS) play a vital role in protecting digital assets in the purview of network security. Intrusion detection systems data are imbalanced and high dimensioned, affecting models’ performance in classifying malicious traffic. This paper uses a denoising autoencoder (DAE) for feature selection to reduce data dimension. To balance the data, the authors use a combined approach of oversampling technique, adaptive synthetic (ADASYN) and a cluster-based under-sampling method using a clustering algorithm, Kmeans. Then, a one-dimensional convolutional neural network (1D-CNN) is used to perform classification. The performance of the proposed model is evaluated on UNSW-NB15 and NSL-KDD datasets. The experimental results show that the model produces a detection rate of 98.79% and 97.23% on UNSW-NB15 for binary classification and multiclass classification, respectively. In the evaluation using NSL-KDD, the model yields a detection rate of 98.52% for binary type classification and 98.16% for multiclass type classification.

Honeyboost: Boosting honeypot performance with data fusion and anomaly detection

With insider attacks becoming more common and costing organizations more every year, it has never been more crucial to be able to predict when an insider attack may happen. Network Anomaly Detection Systems (NADS) have the ability to identify unusual behavior making them useful in predicting cyberattacks, but often suffer from high false positive rates. Honeypots used in conjunction with NADS can help with learning attack behaviors and enable better prediction. However both honeypots and legacy NADS are generally deployed at the gateway to a network.In this paper, we introduce a novel framework called Honeyboost, deployed in the LAN, rather than at the perimeter, focusing on internal-LAN traffic to predict, for example, a malicious insider attack. Honeyboost incorporates an NADS, as well as a honeypot, within an internal network. The anomaly detection technique incorporated into Honeyboost is Lookout, a new method built on extreme value theory that makes it possible to achieve highly desirable low false positive rates. Using data from the LAN Security Monitoring Project, Honeyboost identifies most anomalous nodes before they access the honeypot, thus aiding early detection and prediction.Honeyboost is an unsupervised method comprising of two approaches: horizontal and vertical. The horizontal approach constructs a time series from the communications, via ARP, TCP or UDP, of each node, with node-level features encapsulating the node’s behavior over time. The vertical approach finds anomalies in each protocol (ARP, TCP, UDP) space. Using a window-based model, typical in online scenarios, the horizontal and vertical approaches are combined to identify anomalies and gain useful insights. Experimental results indicate the efficacy of our framework in identifying suspicious activities of nodes.

A Review on Machine Learning Approaches for Network Malicious Behavior Detection in Emerging Technologies.

Network anomaly detection systems (NADSs) play a significant role in every network defense system as they detect and prevent malicious activities. Therefore, this paper offers an exhaustive overview of different aspects of anomaly-based network intrusion detection systems (NIDSs). Additionally, contemporary malicious activities in network systems and the important properties of intrusion detection systems are discussed as well. The present survey explains important phases of NADSs, such as pre-processing, feature extraction and malicious behavior detection and recognition. In addition, with regard to the detection and recognition phase, recent machine learning approaches including supervised, unsupervised, new deep and ensemble learning techniques have been comprehensively discussed; moreover, some details about currently available benchmark datasets for training and evaluating machine learning techniques are provided by the researchers. In the end, potential challenges together with some future directions for machine learning-based NADSs are specified.

Machine Learning Based Network Anomaly Detection

Network Anomaly Detection Systems (NADSs) play prominent role in network security. Due to dynamic change of malware in network traffic data, traditional tools and techniques are failing to protect networks from attack penetration. In this paper we propose a two-phase model to detect and categorize anomalies. First, we selected Random Forest based on the highest accuracy-score out of eleven commonly used algorithms tested with the same set of data. The RF is used to detect anomalies and generate an extra feature named “attack-or-not”. Secondly we fed Neural Network with the data having “attack-or-not” feature to differentiate attack categories, which will help treating each type accordingly. The model performance was good, it scored 0.99 for both Precision and Recall in anomaly detection phase and 0.93 for Precision and 0.88 for Recall in attack categorization phase. We used UNSW-NB15 data set in our study.

A holistic review of Network Anomaly Detection Systems: A comprehensive survey

Network Anomaly Detection Systems (NADSs) are gaining a more important role in most network defense systems for detecting and preventing potential threats. The paper discusses various aspects of anomaly-based Network Intrusion Detection Systems (NIDSs). The paper explains cyber kill chain models and cyber-attacks that compromise network systems. Moreover, the paper describes various Decision Engine (DE) approaches, including new ensemble learning and deep learning approaches. The paper also provides more details about benchmark datasets for training and validating DE approaches. Most of NADSs’ applications, such as Data Centers, Internet of Things (IoT), as well as Fog and Cloud Computing, are also discussed. Finally, we present several experimental explanations which we follow by revealing various promising research directions.

A data-driven metric learning-based scheme for unsupervised network anomaly detection

Most network anomaly detection systems (NADSs) rely on the distance between the connections’ feature vectors to identify attacks. Traditional distance metrics are inefficient for these systems as they deal with heterogeneous features of network connections. In this paper, we address a clustering-based NADS employing a data-driven distance metric. This metric is the outcome of a proposed metric learning method, which extracts its required side information from the training samples. The learned transformation matrix maps the connections’ features to a new feature space in which similar and dissimilar connections are more well-separated while the local neighborhood information of the connections’ features is preserved using the Laplacian Eigenmap technique. The proposed NADS is evaluated over the Kyoto 2006+ and NSL-KDD datasets. The experimental results show that it has superior performance in comparison with a recent SVM-clustering based NADS that employs the traditional Euclidean distance function.

Performances of Machine Learning Algorithms for Binary Classification of Network Anomaly Detection System

The rapid growth of technologies might endanger them to various network attacks due to the nature of data which are frequently exchange their data through Internet and large-scale data that need to be handle. Moreover, network anomaly detection using machine learning faced difficulty when dealing the involvement of dataset where the number of labelled network dataset is very few in public and this caused many researchers keep used the most commonly network dataset (KDDCup99) which is not relevant to employ the machine learning (ML) algorithms for a classification. Several issues regarding these available labelled network datasets are discussed in this paper. The aim of this paper to build a network anomaly detection system using machine learning algorithms that are efficient, effective and fast processing. The finding showed that AODE algorithm is performed well in term of accuracy and processing time for binary classification towards UNSW-NB15 dataset.

The evaluation of Network Anomaly Detection Systems: Statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set

ABSTRACTOver the last three decades, Network Intrusion Detection Systems (NIDSs), particularly, Anomaly Detection Systems (ADSs), have become more significant in detecting novel attacks than Signature Detection Systems (SDSs). Evaluating NIDSs using the existing benchmark data sets of KDD99 and NSLKDD does not reflect satisfactory results, due to three major issues: (1) their lack of modern low footprint attack styles, (2) their lack of modern normal traffic scenarios, and (3) a different distribution of training and testing sets. To address these issues, the UNSW-NB15 data set has recently been generated. This data set has nine types of the modern attacks fashions and new patterns of normal traffic, and it contains 49 attributes that comprise the flow based between hosts and the network packets inspection to discriminate between the observations, either normal or abnormal. In this paper, we demonstrate the complexity of the UNSW-NB15 data set in three aspects. First, the statistical analysis of the observations and the attributes are explained. Second, the examination of feature correlations is provided. Third, five existing classifiers are used to evaluate the complexity in terms of accuracy and false alarm rates (FARs) and then, the results are compared with the KDD99 data set. The experimental results show that UNSW-NB15 is more complex than KDD99 and is considered as a new benchmark data set for evaluating NIDSs.

Network anomaly detection system with optimized DS evidence theory.

Network anomaly detection has been focused on by more people with the fast development of computer network. Some researchers utilized fusion method and DS evidence theory to do network anomaly detection but with low performance, and they did not consider features of network—complicated and varied. To achieve high detection rate, we present a novel network anomaly detection system with optimized Dempster-Shafer evidence theory (ODS) and regression basic probability assignment (RBPA) function. In this model, we add weights for each senor to optimize DS evidence theory according to its previous predict accuracy. And RBPA employs sensor's regression ability to address complex network. By four kinds of experiments, we find that our novel network anomaly detection model has a better detection rate, and RBPA as well as ODS optimization methods can improve system performance significantly.

Accuracy improving guidelines for network anomaly detection systems

An unprecedented growth in computer and communication systems in the last two decades has resulted in a proportional increase in the number and sophistication of network attacks. In particular, the number of previously-unseen attacks has increased exponentially in the last few years. Due to the rapidly evolving nature of network attacks, a considerable paradigm shift has taken place in the intrusion detection community. The main focus is now on Network Anomaly Detection Systems (NADSs) which model and flag deviations from normal/benign behavior of a network and can hence detect previously-unseen attacks. Contemporary NADS borrow concepts from a variety of theoretical fields (e.g., Information theory, stochastic and machine learning, signal processing, etc.) to model benign behavior. These NADSs, however, fall short of achieving acceptable performance levels as therefore widespread commercial deployments. Thus, in this paper, we firstly evaluate the performance of eight prominent network-based anomaly detectors under malicious portscan attacks to identify which NADSs perform better than others and why. These NADSs are evaluated on three criteria: accuracy (ROC curves), scalability (with respect to varying normal and attack traffic rates, and deployment points) and detection delay. These criteria are evaluated using two independently collected datasets with complementary strengths. We then propose novel methods and promising guidelines to improve the accuracy and scalability of existing and future anomaly detectors. Experimental analysis of the proposed guidelines is also presented for the proof of concept.