Anomaly-based Network Intrusion Detection Research Articles

Fed-ANIDS: Federated learning for anomaly-based network intrusion detection systems

Fed-ANIDS: Federated learning for anomaly-based network intrusion detection systems

A Literary Review of Pattern Matching Techniques in Network Intrusion Detection

With the exponential growth in devices and services being added to networks, we are also witnessing an increase in the volume and complexity of threats, urging an increased efficiency in network intrusion detection systems which primarily rely on pattern matching to identify malicious activity on the network. In this literary review of pattern matching techniques in network intrusion detection, we explore the limitations and the research carried out in both signature-based and anomaly-based intrusion detection systems to overcome them. It focuses on the performance improvements in signature-based intrusion detection systems achieved through methodologies and technologies like regular expressions, Hyperscan, RE2, Flashtext, a generalized Aho-Corasick algorithm, usage of Bloom filters and payload sampling. It also covers the usage of machine learning techniques, including genetic algorithms, Support Vector Machines (SVM) and Improved Self-Adaptive Bayesian Algorithm (ISABA), which are used to detect anomalous behavior and identify potential threats in a network in anomaly-based network intrusion detection to assist the security analysts carry out their job functions. Additionally, this review explores the integration of the MITRE ATT&CK framework and Security Information and Event Management (SIEM) systems in network intrusion detection as this framework provides a structured and standardized approach for analyzing the tactics and techniques used by attackers to classify them, while SIEM systems enable the correlation of threat activity across multiple sources, allowing for a more comprehensive and accurate view of the network security. Overall, this literary review provides insights into the state-of-the-art techniques and frameworks used in Network Intrusion Detection based on Pattern Matching, highlighting the significant improvements in performance and detection capabilities.

Recursive Feature Elimination with Cross-Validation with Decision Tree: Feature Selection Method for Machine Learning-Based Intrusion Detection Systems

The frequency of cyber-attacks on the Internet of Things (IoT) networks has significantly increased in recent years. Anomaly-based network intrusion detection systems (NIDSs) offer an additional layer of network protection by detecting and reporting the infamous zero-day attacks. However, the efficiency of real-time detection systems relies on several factors, including the number of features utilized to make a prediction. Thus, minimizing them is crucial as it implies faster prediction and lower storage space. This paper utilizes recursive feature elimination with cross-validation using a decision tree model as an estimator (DT-RFECV) to select an optimal subset of 15 of UNSW-NB15’s 42 features and evaluates them using several ML classifiers, including tree-based ones, such as random forest. The proposed NIDS exhibits an accurate prediction model for network flow with a binary classification accuracy of 95.30% compared to 95.56% when using the entire feature set. The reported scores are comparable to those attained by the state-of-the-art systems despite decreasing the number of utilized features by about 65%.

Effective network intrusion detection using stacking-based ensemble approach

The increasing demand for communication between networked devices connected either through an intranet or the internet increases the need for a reliable and accurate network defense mechanism. Network intrusion detection systems (NIDSs), which are used to detect malicious or anomalous network traffic, are an integral part of network defense. This research aims to address some of the issues faced by anomaly-based network intrusion detection systems. In this research, we first identify some limitations of the legacy NIDS datasets, including a recent CICIDS2017 dataset, which lead us to develop our novel dataset, CIPMAIDS2023-1. Then, we propose a stacking-based ensemble approach that outperforms the overall state of the art for NIDS. Various attack scenarios were implemented along with benign user traffic on the network topology created using graphical network simulator-3 (GNS-3). Key flow features are extracted using cicflowmeter for each attack and are evaluated to analyze their behavior. Several different machine learning approaches are applied to the features extracted from the traffic data, and their performance is compared. The results show that the stacking-based ensemble approach is the most promising and achieves the highest weighted F1-score of 98.24%.

Machine learning to improve the performance of anomaly-based network intrusion detection in big data

With the rapid growth of digital technology communications are overwhelmed by network data traffic. The demand for the internet is growing every day in today's cyber world, raising concerns about network security. Big Data are a term that describes a vast volume of complicated data that is critical for evaluating network patterns and determining what has occurred in the network. Therefore, detecting attacks in a large network is challenging. Intrusion detection system (IDS) is a promising cybersecurity research field. In this paper, we proposed an efficient classification scheme for IDS, which is divided into two procedures, on the CSE-CIC-IDS-2018 dataset, data pre-processing techniques including under-sampling, feature selection, and classifier algorithms were used to assess and decide the best performing model to classify invaders. We have implemented and compared seven classifier machine learning algorithms with various criteria. This work explored the application of the random forest (RF) for feature selection in conjunction with machine learning (ML) techniques including linear regression (LR), k-Nearest Neighbor (k-NN), classification and regression trees (CART), Bayes, RF, multi layer perceptron (MLP), and XGBoost in order to implement IDSS. The experimental results show that the MLP algorithm in the most successful with best performance with evaluation matrix.

Anomaly-based Network Intrusion Detection System using Deep Intelligent Technique

Background and objectives: Computer systems and network infrastructures are still exposed to many security risks and cyber-attack vulnerabilities despite advancements of information security. Traditional signature-based intrusion detection systems and security solutions by matching rule-based mechanism and prior knowledge are insufficient of fully protecting computer networks against novel attacks. For this purpose, Anomaly-based Network Intrusion Detection System (A-NIDS) as cyber security tool is considered for identifying and detecting anomalous behavior in the flow-based network traffic alongside with firewalls and other security measures. The main objective of the research is to improve the detection rate and reduce false-positive rates of the classifier using anomaly-based technique. Methods: an intelligent technique using deep learning algorithm and mutual information feature selection (MIFS) method to select optimal features on the benchmark datasets. Proposed method accurately capable of classifying normal and anomalous states of the data packets in a comprehensive way by combination of Long-Short term memory (LSTM) algorithm and MIFS method. Results: The model achieved encouraging results in terms of accuracy 99.79%, 0.002 false-positive rate with minimum time compared to other models recorded only 81.75s on CSE-CIC-IDS2018 dataset. At the end of the study, comparative studies are conducted to verify the effectiveness of proposed method on three realistic and latest intrusion detection datasets, named CSE_CIC-IDS2018, CIC-IDS2017, and NF-CSE-CIC-IDS2018 dataset. Conclusions: Proposed model in a combination of LSTM NN and Feature selection method (MIFS) increased detection rate and reduced false-positive alarms, also the model able to detect low frequent attacks while other existing models are suffering from.

Detection of Network Attacks using Machine Learning and Deep Learning Models

Anomaly-based network intrusion detection systems are highly significant in detecting network attacks. Robust machine learning and deep learning models for identifying network intrusion and attack types are proposed in this paper. Proposed models have experimented with the UNSW-NB15 dataset of 49 features for nine different attack samples. Decision Tree classifier produced the best accuracy of 99.05% compared to ensemble models - Random forest(98.96%), Adaboost(97.87%), and XGBoost(98.08%). K-Nearest Neighbour classifier trained for various values of K and best performance obtained for K=7 with the accuracy of 95.58%. A Deep Learning model with two dense layers with ReLU activation and a third dense layer with a Sigmoid activation function is designed for binary classification and produced good accuracy of 98.44% with ADAM optimizer, 80:20 Train-Test Split Ratio. Network attack exploits are detected with an accuracy 95% by XGBoost, Fuzzers attack with 90% accuracy by Random Forest, Generic attacks with 99% accuracy by Random Forest, and Reconnaissance attacks with 79% by Decision Trees. All features are relevant and strong in network attack detection, which eliminates the requirement of feature selection.

D-Score: An expert-based method for assessing the detectability of IoT-related cyber-attacks

IoT devices are known to be vulnerable to various cyber-attacks, such as data exfiltration and the execution of flooding attacks as part of a DDoS attack. When it comes to detecting such attacks using network traffic analysis, it has been shown that some attack scenarios are not always equally easy to detect if they involve different IoT models. That is, when targeted at some IoT models, a given attack can be detected rather accurately, while when targeted at others the same attack may result in too many false alarms. In this research, we attempt to explain this variability of IoT attack detectability and devise a risk assessment method capable of addressing a key question: how easy is it for an anomaly-based network intrusion detection system to detect a given cyber-attack involving a specific IoT model? In the process of addressing this question we (a) investigate the predictability of IoT network traffic, (b) present a novel taxonomy for IoT attack detection which also encapsulates traffic predictability aspects, (c) propose an expert-based attack detectability estimation method which uses this taxonomy to derive a detectability score (termed ‘D-Score’) for a given combination of IoT model and attack scenario, and (d) empirically evaluate our method while comparing it with a data-driven method.

Context adaptive ensemble classification mechanism with multi‐criteria decision making for network intrusion detection

SummaryNowadays, intrusion detection systems (IDSs) enabled with computational intelligence for electing the suitable classifiers to learn the patterns of various types of network attacks are under study. Selection mechanism of single or ensemble classifiers based on performance and ranking is to be enabled in the IDS framework to make accurate pattern predictions. A new anomaly based network intrusion detection system framework is proposed using context adaptive classification through the technique for order of preference by similarity to ideal solution (TOPSIS) ranking mechanism with hierarchy based chi‐square and bat algorithms for feature selection. Parameters like accuracy, false positive rate (FPR), and classification model building time are explored for choosing the best and situation aware classifiers using TOPSIS. For experimentation, NSL‐KDD and UNSW‐NB15 datasets are used. The proposed system opts decision tree (DT) algorithm for NSL‐KDD and produces 98.77% accuracy, 0.03% FPR with 8 features. Ensemble learner is selected for the UNSW‐NB15 dataset using DT and support vector machine classifiers with 9 features. Combined classifier predicts 89.43% accuracy and 3.215% FPR. Experimentation on the proposed methodology with the state of the art approaches produces promising results.

N-GAN: a novel anomaly-based network intrusion detection with generative adversarial networks

N-GAN: a novel anomaly-based network intrusion detection with generative adversarial networks

A comparative study using supervised learning for anomaly detection in network traffic

A user connects to hundreds of remote networks daily, some of which can be corrupted by malicious sources. To overcome this problem, a variety of Network Intrusion Detection systems are built, which aim to detect harmful networks before they establish a connection with the user’s local system. This paper focuses on proposing a model for Anomaly based Network Intrusion Detection systems (NIDS), by performing comparisons of various Supervised Learning Algorithms on metric of their accuracy. Two datasets were used and analysed, each having different properties in terms of the volume of data they contain and their use cases. Feature engineering was done to retrieve the most optimum features of both the datasets and only the top 25% best features were used to build the models – a smaller subset of features not only aids in decreasing the capital required to collect the data but also gets rid of redundant and noisy information. Two different splicing methods were used to train the data and each method showed different trends on the ML models.

Toward A Holistic, Efficient, Stacking Ensemble Intrusion Detection System using a Real Cloud-based Dataset

Network intrusion detection is a key step in securing today’s constantly developing networks. Various experiments have been put forward to propose new methods for resisting harmful cyber behaviors. Though, as cyber-attacks turn out to be more complex, the present methodologies fail to adequately solve the problem. Thus, network intrusion detection is now a significant decision-making challenge that requires an effective and intelligent approach. Various machine learning algorithms such as decision trees, neural networks, K nearest neighbor, logistic regression, support vector machine, and Naive Bayes have been utilized to detect anomalies in network traffic. However, such algorithms require adequate datasets to train and evaluate anomaly-based network intrusion detection systems. This paper presents a testbed that could be a model for building real-world datasets, as well as a newly generated dataset, derived from real network traffic, for intrusion detection. To utilize this real dataset, the paper also presents an ensemble intrusion detection model using a meta-classification approach enabled by stacked generalization to address the issue of detection accuracy and false alarm rate in intrusion detection systems.

A System to automate the development of anomaly-based network intrusion detection model

Cyber security is the major concern in today’s world. Over the past couple of decades, the internet has grown to such an extent that almost every individual living on this planet has the access to the internet today. This can be viewed as one of the major achievements in the human race, but on the flip side of the coin, this gave rise to a lot of security issues for every individual or the company that is accessing the web through the internet. Hackers have become active and are always monitoring the networks to grab every possible opportunity to attack a system and make the best fortune out of its vulnerabilities. To safeguard people’s and organization’s privacy in this cyberspace, different network intrusion detection systems have been developed to detect the hacker’s presence in the networks. These systems fall under signature based and anomaly based intrusion detection systems. This paper deals with using anomaly based intrusion detection technique to develop an automation system to both train and test supervised machine learning models, which is developed to classify real time network traffic as to whether it is malicious or not. Currently the best models by considering both detection success rate and the false positives rate are Artificial Neural Networks(ANN) followed by Support Vector Machines(SVM). In this paper, it is verified that Artificial Neural Network (ANN) based machine learning with wrapper feature selection outperforms support vector machine (SVM) technique while classifying network traffic as harmful or harmless. Initially to evaluate the performance of the system, NSL-KDD dataset is used to train and test the SVM and ANN models and finally classify real time network traffic using these models. This system can be used to carry out model building automatically on the new datasets and also for classifying the behaviour of the provided dataset without having to code.

Evaluating and Improving Adversarial Robustness of Machine Learning-Based Network Intrusion Detectors

Machine learning (ML), especially deep learning (DL) techniques have been increasingly used in anomaly-based network intrusion detection systems (NIDS). However, ML/DL has shown to be extremely vulnerable to adversarial attacks, especially in such security-sensitive systems. Many adversarial attacks have been proposed to evaluate the robustness of ML-based NIDSs. Unfortunately, existing attacks mostly focused on feature-space and/or white-box attacks, which make impractical assumptions in real-world scenarios, leaving the study on practical gray/black-box attacks largely unexplored. To bridge this gap, we conduct the first systematic study of the gray/black-box traffic-space adversarial attacks to evaluate the robustness of ML-based NIDSs. Our work outperforms previous ones in the following aspects: (i) practical-the proposed attack can automatically mutate original traffic with extremely limited knowledge and affordable overhead while preserving its functionality; (ii) generic-the proposed attack is effective for evaluating the robustness of various NIDSs using diverse ML/DL models and non-payload-based features; (iii) explainable-we propose an explanation method for the fragile robustness of ML-based NIDSs. Based on this, we also propose a defense scheme against adversarial attacks to improve system robustness. We extensively evaluate the robustness of various NIDSs using diverse feature sets and ML/DL models. Experimental results show our attack is effective (e.g., >97% evasion rate in half cases for Kitsune, a state-of-the-art NIDS) with affordable execution cost and the proposed defense method can effectively mitigate such attacks (evasion rate is reduced by >50% in most cases).

CBAM: A Contextual Model for Network Anomaly Detection

Anomaly-based intrusion detection methods aim to combat the increasing rate of zero-day attacks, however, their success is currently restricted to the detection of high-volume attacks using aggregated traffic features. Recent evaluations show that the current anomaly-based network intrusion detection methods fail to reliably detect remote access attacks. These are smaller in volume and often only stand out when compared to their surroundings. Currently, anomaly methods try to detect access attack events mainly as point anomalies and neglect the context they appear in. We present and examine a contextual bidirectional anomaly model (CBAM) based on deep LSTM-networks that is specifically designed to detect such attacks as contextual network anomalies. The model efficiently learns short-term sequential patterns in network flows as conditional event probabilities. Access attacks frequently break these patterns when exploiting vulnerabilities, and can thus be detected as contextual anomalies. We evaluated CBAM on an assembly of three datasets that provide both representative network access attacks, real-life traffic over a long timespan, and traffic from a real-world red-team attack. We contend that this assembly is closer to a potential deployment environment than current NIDS benchmark datasets. We show that, by building a deep model, we are able to reduce the false positive rate to 0.16% while effectively detecting six out of seven access attacks, which is significantly lower than the operational range of other methods. We further demonstrate that short-term flow structures remain stable over long periods of time, making the CBAM robust against concept drift.

A Convolutional Neural Network for Improved Anomaly-Based Network Intrusion Detection.

Cybersecurity protects and recovers computer systems and networks from cyber attacks. The importance of cybersecurity is growing commensurately with people's increasing reliance on technology. An anomaly detection-based network intrusion detection system is essential to any security framework within a computer network. In this article, we propose two models based on deep learning to address the binary and multiclass classification of network attacks. We use a convolutional neural network architecture for our models. In addition, a hybrid two-step preprocessing approach is proposed to generate meaningful features. The proposed approach combines dimensionality reduction and feature engineering using deep feature synthesis. The performance of our models is evaluated using two benchmark data sets, namely the network security laboratory-knowledge discovery in databases data set and the University of New South Wales Network Based 2015 data set. The performance is compared with similar deep learning approaches in the literature, as well as state-of-the-art classification models. Experimental results show that our models achieve good performance in terms of accuracy and recall, outperforming similar models in the literature.

A Review on Machine Learning Approaches for Network Malicious Behavior Detection in Emerging Technologies.

Network anomaly detection systems (NADSs) play a significant role in every network defense system as they detect and prevent malicious activities. Therefore, this paper offers an exhaustive overview of different aspects of anomaly-based network intrusion detection systems (NIDSs). Additionally, contemporary malicious activities in network systems and the important properties of intrusion detection systems are discussed as well. The present survey explains important phases of NADSs, such as pre-processing, feature extraction and malicious behavior detection and recognition. In addition, with regard to the detection and recognition phase, recent machine learning approaches including supervised, unsupervised, new deep and ensemble learning techniques have been comprehensively discussed; moreover, some details about currently available benchmark datasets for training and evaluating machine learning techniques are provided by the researchers. In the end, potential challenges together with some future directions for machine learning-based NADSs are specified.

Anomaly-based network intrusion detection with ensemble classifiers and meta-heuristic scale (ECMHS) in traffic flow streams

The exponential growth in the internet services lead to enormous growth in the network traffic. As the services are increasing the numbers of network attacks are also gradually increasing. From the contemporary literature it is proved that machine learning techniques have gained importance in addressing security issues in networks and these techniques rely on features and its values to extract the knowledge. It is evidenced that phenomenal growth in the volume of transactions leads to deviation in feature values. Hence, it is necessary to consider the associability among the transactions and its feature values. In this paper, a Meta-heuristic association scale is proposed to derive a threshold value for the transaction and further, an ensemble classifier is used to analyse the transaction as normal or attack. Ensemble classifier used in the proposed system is based on drift detection which has the ability to analyze the requests at stream level. The proposed model derives the features from the stream level and uses drift detection to analyze the stream characteristics. The experimental study is carried out on the benchmark data to analyze the statistical parameters accuracy, false alarm rate, positive predictive values. Moreover, the ECMHS is compared with the other benchmark models depicted in contemporary literature.

Training Guidance with KDD Cup 1999 and NSL-KDD Data Sets of ANIDINR: Anomaly-Based Network Intrusion Detection System

Training Guidance with KDD Cup 1999 and NSL-KDD Data Sets of ANIDINR: Anomaly-Based Network Intrusion Detection System

Training Guidance with KDD Cup 1999 and NSL-KDD Data Sets of ANIDINR: Anomaly-Based Network Intrusion Detection System

Abstract In today’s world, the protection of the computer networks remains one of the most crucial and difficult challenges in cyber security. In this work, a passive defence system ANIDINR is presented, aiming to monitor and protect computer networks. Our effort is focused on providing step-by-step guidance on methodologies selection and execution for the Machine and Deep Learning models’ training. Taking as an input two data sets, five MDL models are evaluated. Our goals are to minimise the percentage of Undetected Attack, the percentage of False Alarm Rate and the overall testing time. Based on this set-up, the proposed system is capable to predict in near-to-real time well-known and zero-day computer network attacks.