Flow-based Intrusion Detection Research Articles

Correction: Flow-based intrusion detection on software-defined networks: a multivariate time series anomaly detection approach

Correction: Flow-based intrusion detection on software-defined networks: a multivariate time series anomaly detection approach

Investigating on the robustness of flow-based intrusion detection system against adversarial samples using Generative Adversarial Networks

Recently, Software Defined Networking (SDN) has emerged as the key technology in programming and orchestrating security policy in the security operations centers (SOCs) for heterogeneous networks. Typically, machine learning-based intrusion detection systems (ML-IDS) have been deployed and associated with SDN to leverage the features of a programmable network to defend against sophisticated cyberattacks in anomaly detection. Unfortunately, such ML-based IDSs are easily vulnerable to adversarial attacks due to the lack of diverse forms of malicious records in the training dataset. The missing data sample in the training phase can lead to a lower detection rate in real-world scenarios with adversarial settings. In this paper, we explore the ability of Wasserstein Generative Adversarial Networks with Gradient Penalty (WGAN-GP), WGAN-GP with two timescale update rule (WGAN-GP TTUR), and AdvGAN in generating perturbed attack samples to bypass attack detectors. Then, this approach is used to continuously evaluate the robustness of ML-based IDSs and then upgrade them as a service in SDN. The experimental results on CICIDS2018 and InSDN datasets demonstrate that generated adversarial samples can be used to fool targeted IDS. Later, those created samples can supplement the original ones in retraining IDS to improve the resilience of the attack detector.

Flow-based intrusion detection on software-defined networks: a multivariate time series anomaly detection approach

In this study, we present and implement the SAnDet (SDN anomaly detector) architecture, an anomaly-based intrusion detection system designed to take advantage of the capabilities offered by software-defined networking (SDN) architecture, as a controller application. The SAnDet system is composed of three modules: statistics collection, anomaly detection, and anomaly prevention. In particular, we utilize replicator neural networks (RNN), which is a specialized variant of the autoencoder, and the LSTM-based encoder–decoder (EncDecAD) method, which is a special type of long short-term memory (LSTM) network that has demonstrated a strong performance on data series particularly, to identify unknown attacks using flow features collected from OpenFlow switches. In our experiments, we utilize flow-based features extracted from network traffic data containing various types of attacks as input to our models in the form of time series. We evaluate the performance of our methods using the accuracy and area under the receiver operating characteristic curve (AUC) metrics. Our experimental results demonstrate that EncDecAD outperforms RNN and that our approach offers several benefits over previously conducted research.

Flow-based intrusion detection system in Vehicular Ad hoc Network using context-aware feature extraction

Autonomous transportation systems have an immense impact on the way of our day-to-day commute or traveling. Connectedness is an inseparable part of these systems, and it is possible through Intelligent Transportation Systems applications. These applications use a network system known as Vehicular Ad hoc Network (VANET). However, the safety that is provided by VANETs can be easily compromised by malicious users. Hence there is a need for an Intrusion Detection System (IDS). We designed an IDS model that can collect network data cooperatively from vehicles and Roadside Units (RSUs). For training the core of our proposed IDS we generated synthetic network data with the help of the widely used Network Simulator 3 (ns-3) and Simulation of Urban Mobility (SUMO) simulation tools. We also generated separate test data that is not just a split part of the training data. This ensures the elimination of the misleading performance result of an overfitted model. We employed a multi-class IDS using Convolutional Neural Network (CNN) with a novel feature extraction method known as Context-Aware Feature Extraction-Based CNN (CAFECNN). The CAFECNN model takes advantage of the collected network flow data to detect both passive and active types of attacks. The results show that the proposed model is stronger in identification of hard-to-detect passive attacks compared to traditional machine learning methods. We also evaluated the model using real-time simulations and observed an immediate improvement in the performance of the network despite the intruders, especially in terms of packet delivery ratio and network throughput.

Evaluation of Machine Learning Techniques for Traffic Flow-Based Intrusion Detection

Cybersecurity is one of the great challenges of today's world. Rapid technological development has allowed society to prosper and improve the quality of life and the world is more dependent on new technologies. Managing security risks quickly and effectively, preventing, identifying, or mitigating them is a great challenge. The appearance of new attacks, and with more frequency, requires a constant update of threat detection methods. Traditional signature-based techniques are effective for known attacks, but they are not able to detect a new attack. For this reason, intrusion detection systems (IDS) that apply machine learning (ML) techniques represent an alternative that is gaining importance today. In this work, we have analyzed different machine learning techniques to determine which ones permit to obtain the best traffic classification results based on classification performance measurements and execution times, which is decisive for further real-time deployments. The CICIDS2017 dataset was selected in this work since it contains bidirectional traffic flows (derived from traffic captures) that include benign traffic and different types of up-to-date attacks. Each traffic flow is characterized by a set of connection-related attributes that can be used to model the traffic and distinguish between attacks and normal flows. The CICIDS2017 also contains the raw network traffic captures collected during the dataset creation in a packet-based format, thus permitting to extract the traffic flows from them. Various classification techniques have been evaluated using the Weka software: naive Bayes, logistic, multilayer perceptron, sequential minimal optimization, k-nearest neighbors, adaptive boosting, OneR, J48, PART, and random forest. As a general result, methods based on decision trees (PART, J48, and random forest) have turned out to be the most efficient with F1 values above 0.999 (average obtained in the complete dataset). Moreover, multiclass classification (distinguishing between different types of attack) and binary classification (distinguishing only between normal traffic and attack) have been compared, and the effect of reducing the number of attributes using the correlation-based feature selection (CFS) technique has been evaluated. By reducing the complexity in binary classification, better results can be obtained, and by selecting a reduced set of the most relevant attributes, less time is required (above 30% of decrease in the time required to test the model) at the cost of a small performance loss. The tree-based techniques with CFS attribute selection (six attributes selected) reached F1 values above 0.990 in the complete dataset. Finally, a conventional tool like Zeek has been used to process the raw traffic captures to identify the traffic flows and to obtain a reduced set of attributes from these flows. The classification results obtained using tree-based techniques (with 14 Zeek-based attributes) were also very high, with F1 above 0.997 (average obtained in the complete dataset) and low execution times (allowing several hundred thousand flows/s to be processed). These classification results obtained on the CICIDS2017 dataset allow us to affirm that the tree-based machine learning techniques may be appropriate in the flow-based intrusion detection problem and that algorithms, such as PART or J48, may offer a faster alternative solution to the RF technique.

Ensemble of Bio-inspired Algorithm with Statistical Measures for Feature Selection to Design a Flow-Based Intrusion Detection System

In today's high-speed network, the existing Intrusion Detection System (IDS) approaches experience more false alarm rates with low detection capability. Nowadays, IDS needs to analyze a considerable amount of data. The larger the amount of data results in the longer the time to analyze it, which delays attack detection. The IDS usability is defined as its capability to trigger an alarm early enough to minimize the damage that an ongoing attack can cause and provide a reduced range of warning (false alarm). These underline the necessity of feature selection in IDS to identify the informative features and overlook the irrelevant or redundant features that affect the IDS's detection rate and computational complexity. It implies that anticipating an ideal number of features from a flow-based intrusion dataset can improve IDS accuracy. Therefore, this paper proposes an ensemble of a bio-inspired algorithm (Krill Herd Algorithm) with statistical measures (Information Gain) to select optimal features for a flow-based IDS. This ensemble technique has shown improvement in the detection rate, decreases the false alarm rate, and reduces the computation time of the IDS.

Hybrid intrusion detection system based on Dempster-Shafer evidence theory

Cyber-attacks are becoming increasingly sophisticated, posing greater challenges in accurately detecting intrusions. Failure to prevent intrusions could degrade the credibility of security services. Intrusion Detection System (IDS) is one of the most effective paradigms to identify attack behaviors. This paper proposes a novel hybrid intrusion detection system called DST-IDS. The proposed method employs both packet-based and flow-based intrusion detection techniques and combines them with Dempster-Shafer Theory (DST). DST-IDS has an ensemble-like framework. It takes both traffic flows and their first N packets as inputs; flow-based IDS aims to predict traffic flows and packet-based IDS detects attacks in the corresponding packets; DST is then applied to fuse predictions of flow-based IDS and packet-based IDS to a final detection result. We also design a novel data collection/processing tool in DST-IDS to reduce the data volume required to perform intrusion detection and enable early detection. In addition, DST-IDS is designed to work with heterogeneous data distribution where the distribution of the training dataset can differ from the data distribution during implementation. This property drastically improves the practicality of DST-IDS. We run experiments on public datasets and real networks to evaluate the proposed method. The experimental results show that DST-IDS outperforms state-of-the-art benchmarks in terms of intrusion detection accuracy and detection speed. Particularly, DST-IDS provides real-time detection in real networks and handles well heterogeneous data distribution.

IDS for Industrial Applications: A Federated Learning Approach with Active Personalization.

Internet of Things (IoT) is a concept adopted in nearly every aspect of human life, leading to an explosive utilization of intelligent devices. Notably, such solutions are especially integrated in the industrial sector, to allow the remote monitoring and control of critical infrastructure. Such global integration of IoT solutions has led to an expanded attack surface against IoT-enabled infrastructures. Artificial intelligence and machine learning have demonstrated their ability to resolve issues that would have been impossible or difficult to address otherwise; thus, such solutions are closely associated with securing IoT. Classical collaborative and distributed machine learning approaches are known to compromise sensitive information. In our paper, we demonstrate the creation of a network flow-based Intrusion Detection System (IDS) aiming to protecting critical infrastructures, stemming from the pairing of two machine learning techniques, namely, federated learning and active learning. The former is utilized for privately training models in federation, while the latter is a semi-supervised approach applied for global model adaptation to each of the participant’s traffic. Experimental results indicate that global models perform significantly better for each participant, when locally personalized with just a few active learning queries. Specifically, we demonstrate how the accuracy increase can reach 7.07% in only 10 queries.

Flow‐based intrusion detection algorithm for supervisory control and data acquisition systems: A real‐time approach

Intrusion detection in supervisory control and data acquisition (SCADA) systems is integral because of the critical roles of these systems in industries. However, available approaches in the literature lack representative flow-based datasets and reliable real-time adaption and evaluation. A publicly available labelled dataset to support flow-based intrusion detection research specific to SCADA systems is presented. Cyberattacks were carried out against our SCADA system test bed to generate this flow-based dataset. Moreover, a flow-based intrusion detection system (IDS) is developed for SCADA systems using a deep learning algorithm. We used the dataset to develop this IDS model for real-time operations of SCADA systems to detect attacks momentarily after they happen. The results show empirical proof of the model’s adequacy when deployed online to detect cyberattacks in real time.

A flow-based intrusion detection framework for internet of things networks

The application of the Internet of Things concept in domains such as industrial control, building automation, human health, and environmental monitoring, introduces new privacy and security challenges. Consequently, traditional implementation of monitoring and security mechanisms cannot always be presently feasible and adequate due to the number of IoT devices, their heterogeneity and the typical limitations of their technical specifications. In this paper, we propose an IP flow-based Intrusion Detection System (IDS) framework to monitor and protect IoT networks from external and internal threats in real-time. The proposed framework collects IP flows from an IoT network and analyses them in order to monitor and detect attacks, intrusions, and other types of anomalies at different IoT architecture layers based on some flow features instead of using packet headers fields and their payload. The proposed framework was designed to consider both the IoT network architecture and other IoT contextual characteristics such as scalability, heterogeneity, interoperability, and the minimization of the use of IoT networks resources. The proposed IDS framework is network-based and relies on a hybrid architecture, as it involves both centralized analysis and distributed data collection components. In terms of detection method, the framework uses a specification-based approach drawn on normal traffic specifications. The experimental results show that this framework can achieve ≈ 100% success and 0% of false positives in detection of intrusions and anomalies. In terms of performance and scalability in the operation of the IDS components, we study and compare it with three different conventional IDS (Snort, Suricata, and Zeek) and the results demonstrate that the proposed solution can consume fewer computational resources (CPU, RAM, and persistent memory) when compared to those conventional IDS.

Abstracting Packet Header Information for Intrusion Detection in High-Speed Networks

Increase in network traffic coupled with increasing adoption of end-to-end encryption of network packets are two major factors threatening the potency, or even the relevance, of packet-based intrusion detection techniques. Also, end-to-end encryption makes it nearly impossible for network and host-based intrusion detection system to analyze traffic for potential threats and intrusion, hence, the need for an alternative approach. Flow-based intrusion detection system has been proposed as an alternative to a packet-based intrusion detection system as it relies on information embedded in packet header and various statistical analyses of network flow for detecting intrusion. This paper proposes packet header information abstraction model for intrusion detection on the UNSW-NB15 intrusion dataset. Four existing classification algorithms which include: Classification and Regression Tree (CART), Naïve Bayes (NB), K-Nearest Neighbour (KNN), and Support Vector Machine (SVM) are used to evaluate the degree of representativeness of the proposed model using accuracy, sensitivity and specificity evaluation metrics. An average accuracy of 97.95% was recorded across the four models with the minimum accuracy of 97.76 on SVM and best accuracy of 98.05% on CART while Sensitivity of 1.0 on both CART and NB shows that the model performs well in correctly identifying attacks in the network. The average specificity of 0.98 is also an indication of low false positive. Results obtained show that the proposed abstraction model achieves high accuracy, sensitivity and specificity. The model can be used as filter on a high-speed network whereby packets flagged as an attack can be subjected to further analysis.Keywords—Data Abstraction, Data Mining,Flow-based, Intrusion detection, Network Security

An effective convolutional neural network based on SMOTE and Gaussian mixture model for intrusion detection in imbalanced dataset

Network Intrusion Detection System (NIDS) is a key security device in modern networks to detect malicious activities. However, the problem of imbalanced class associated with intrusion detection dataset limits the classifier’s performance for minority classes. To improve the detection rate of minority classes while ensuring efficiency, we propose a novel class imbalance processing technology for large-scale dataset, referred to as SGM, which combines Synthetic Minority Over-Sampling Technique (SMOTE) and under-sampling for clustering based on Gaussian Mixture Model (GMM). We then design a flow-based intrusion detection model, SGM-CNN, which integrates imbalanced class processing with convolutional neural network, and investigate the impact of different numbers of convolution kernels and different learning rates on model performance. The advantages of the proposed model are verified using the UNSW-NB15 and CICIDS2017 datasets. The experimental results show that i) for binary classification and multiclass classification on the UNSW-NB15 dataset, SGM-CNN achieves a detection rate of 99.74% and 96.54%, respectively; ii) for 15-class classification on the CICIDS2017 dataset, it achieves a detection rate of 99.85%. We compare five imbalanced processing methods and two classification algorithms, and conclude that SGM-CNN provides an effective solution to imbalanced intrusion detection and outperforms the state-of-the-art intrusion detection methods.

An intelligent flow-based and signature-based IDS for SDNs using ensemble feature selection and a multi-layer machine learning-based classifier

Software-defined networking is a new paradigm that overcomes problems associated with traditional network architecture by separating the control logic from data plane devices. It also enhances performance by providing a highly-programmable interface that adapts to dynamic changes in network policies. As software-defined networking controllers are prone to single-point failures, providing security is one of the biggest challenges in this framework. This paper intends to provide an intrusion detection mechanism in both the control plane and data plane to secure the controller and forwarding devices respectively. In the control plane, we imposed a flow-based intrusion detection system that inspects every new incoming flow towards the controller. In the data plane, we assigned a signature-based intrusion detection system to inspect traffic between Open Flow switches using port mirroring to analyse and detect malicious activity. Our flow-based system works with the help of trained, multi-layer machine learning-based classifier, while our signature-based system works with rule-based classifiers using the Snort intrusion detection system. The ensemble feature selection technique we adopted in the flow-based system helps to identify the prominent features and hasten the classification process. Our proposed work ensures a high level of security in the Software-defined networking environment by working simultaneously in both control plane and data plane.

Anomaly-Based Intrusion Detection From Network Flow Features Using Variational Autoencoder

The rapid increase in network traffic has recently led to the importance of flow-based intrusion detection systems processing a small amount of traffic data. Furthermore, anomaly-based methods, which can identify unknown attacks are also integrated into these systems. In this study, the focus is concentrated on the detection of anomalous network traffic (or intrusions) from flow-based data using unsupervised deep learning methods with semi-supervised learning approach. More specifically, Autoencoder and Variational Autoencoder methods were employed to identify unknown attacks using flow features. In the experiments carried out, the flow-based features extracted out of network traffic data, including typical and different types of attacks, were used. The Receiver Operating Characteristics (ROC) and the area under ROC curve, resulting from these methods were calculated and compared with One-Class Support Vector Machine. The ROC curves were examined in detail to analyze the performance of the methods in various threshold values. The experimental results show that Variational Autoencoder performs, for the most part, better than Autoencoder and One-Class Support Vector Machine.

Effects of Machine Learning Approach in Flow-Based Anomaly Detection on Software-Defined Networking

Recent advancements in software-defined networking (SDN) make it possible to overcome the management challenges of traditional networks by logically centralizing the control plane and decoupling it from the forwarding plane. Through a symmetric and centralized controller, SDN can prevent security breaches, but it can also bring in new threats and vulnerabilities. The central controller can be a single point of failure. Hence, flow-based anomaly detection system in OpenFlow Controller can secure SDN to a great extent. In this research, we investigated two different approaches of flow-based intrusion detection system in OpenFlow Controller. The first of which is based on machine-learning algorithm where NSL-KDD dataset with feature selection ensures the accuracy of 82% with random forest classifier using the gain ratio feature selection evaluator. In the later phase, the second approach is combined with a deep neural network (DNN)-based intrusion detection system based on gated recurrent unit-long short-term memory (GRU-LSTM) where we used a suitable ANOVA F-Test and recursive feature elimination selection method to boost classifier output and achieve an accuracy of 88%. Substantial experiments with comparative analysis clearly show that, deep learning would be a better choice for intrusion detection in OpenFlow Controller.

A two-stage flow-based intrusion detection model for next-generation networks.

The next-generation network provides state-of-the-art access-independent services over converged mobile and fixed networks. Security in the converged network environment is a major challenge. Traditional packet and protocol-based intrusion detection techniques cannot be used in next-generation networks due to slow throughput, low accuracy and their inability to inspect encrypted payload. An alternative solution for protection of next-generation networks is to use network flow records for detection of malicious activity in the network traffic. The network flow records are independent of access networks and user applications. In this paper, we propose a two-stage flow-based intrusion detection system for next-generation networks. The first stage uses an enhanced unsupervised one-class support vector machine which separates malicious flows from normal network traffic. The second stage uses a self-organizing map which automatically groups malicious flows into different alert clusters. We validated the proposed approach on two flow-based datasets and obtained promising results.

Flow Based Intrusion Detection System Using Multistage Neural Network

With the rapid expansion of computer networks during the past decade, security has become a crucial issue for computer systems. And to keep security at highest level, there is an increasing need for effective security monitors such as Network Intrusion Detection System to prevent such illicit. In the recent years many researchers focus their hard work on this field using different approaches to build dependable intrusion detection systems. One of these approaches is Flow-based intrusion detection systems that rely on aggregated network traffic flows. In this paper, Multistage Neural Network intrusion detection system based on aggregated flow data is proposed for detecting and classifying attacks in network traffic. The proposed system detects significant changes in the traffic that could be a possible attack in the first stage of neural network, while the second stage has the ability to recognize an attack, to differentiate one attack from another i.e. classifying attack, and the most important, to detect new attacks with high detection rate and low false negative. Two different neural network structures with the use of different training algorithms have been used in our proposed Intrusion Detection System. The experimental results show that the designed system is promising in terms of accuracy and low probability of false alarms, where the overall accuracy classification rate average is equal to 99.25%.

Cluster Ensemble with Link-Based Approach for Botnet Detection

Botnet detection is one of the most imminent tasks for cyber security. Among popular botnet countermeasures, an intrusion detection system is the prominent mechanism. In the past, packet-based intrusion detection systems were popular. However, flow-based intrusion detection systems have been preferred in recent years due to their ability to adapt to modern high-speed networks. A collection of flows from an enterprise network usually contains both botnet traffic and normal traffic. To classify this traffic, supervised machine learning algorithms, i.e., classifications, have been applied and achieved a high accuracy. In an effort to improve the ability of intrusion detection systems against botnets, some studies have suggested partitioning flows into clusters before applying the classifications and this step could significantly reduce the complexity of a flow set. However, the instability of individual clustering algorithms is still a constraint for botnet detection.To overcome this bottleneck, we propose a novel method that combines individual partitions to become a strong learner through the use of a link-based algorithm. Our experiments show that our cluster ensemble model outperforms existing botnet detection mechanisms with a high reliability. We also determine the balance between accuracy and computer resources for botnet detection, and thereby propose a range for the maximum duration time of flows in botnet research.

Applying One-Class Classification Techniques to IP Flow Records for Intrusion Detection

Flow-basedintrusiondetectionsystemsanalyzeIPflowrecordstodetectattacksagainst computer networks. IP flow records contain aggregated packet header information; therefore, the amount of data processed by the intrusion detection system is reduced. In addition, since no pay- load is analyzed, the end-to-end encryption does not affect the deployment of intermediate intru- sion detection system. In this paper, we evaluate one-class classification techniques for detection of malicious flows at an initial stage of a multi-stage flow-based intrusion detection system. The initial stage uses minimal flow attributes and only decide if the IP flow is normal or malicious. Since there is only one class of interest (malicious) at the initial stage, we use one-class classifi- cation for detection of malicious flows. In this paper, we review available one-class classification techniques and evaluate them on a flow-based dataset to determine their performance for detec- tion of malicious flows. Our results show that one-class classification techniques using boundary methods give best results in detection of malicious IP flows.

Towards Multi-Stage Intrusion Detection using IP Flow Records

Traditional network-based intrusion detection sys-tems using deep packet inspection are not feasible for modern high-speed networks due to slow processing and inability to read encrypted packet content. As an alternative to packet-based intrusion detection, researchers have focused on flow-based intrusion detection techniques. Flow-based intrusion detection systems analyze IP flow records for attack detection. IP flow records contain summarized traffic information. However, flow data is very large in high-speed networks and cannot be processed in real-time by the intrusion detection system. In this paper, an efficient multi-stage model for intrusion detection using IP flows records is proposed. The first stage in the model classifies the traffic as normal or malicious. The malicious flows are further analyzed by a second stage. The second stage associates an attack type with malicious IP flows. The proposed multi-stage model is efficient because the majority of IP flows are discarded in the first stage and only malicious flows are examined in detail. We also describe the implementation of our model using machine learning techniques.