Signature-based Network Intrusion Detection Systems Research Articles

A perspective–retrospective analysis of diversity in signature-based open-source network intrusion detection systems

The signature-based network intrusion detection systems (IDSs) entail relying on a pre-established signatures and IP addresses that are frequently updated to keep up with the rapidly evolving threat landscape. To effectively evaluate the efficacy of these updates, a comprehensive, long-term assessment of the IDSs’ performance is required. This article presents a perspective–retrospective analysis of the Snort and Suricata IDSs using rules that were collected over a 4-year period. The study examines how these IDSs perform when monitoring malicious traffic using rules from the past, as well as how they behave when monitoring the same traffic using updated rules in the future. To accomplish this, a set of Snort Subscribed and Suricata Emerging Threats rules were collected from 2017 to 2020, and a labeled PCAP data from 2017 to 2018 was analyzed using past and future rules relative to the PCAP date. In addition to exploring the evolution of Snort and Suricata IDSs, the study also analyses the functional diversity that exists between these IDSs. By examining the evolutionary behavior of signature-based IDSs and their diverse configurations, the research provides valuable insights into how their performance can be impacted. These insights can aid security architects in combining and layering IDSs in a defence-in-depth deployment.

Detecting Unseen Anomalies in Network Systems by Leveraging Neural Networks

Despite all the progress achieved in recent years in detecting anomalies in network systems, detecting unseen anomalies such as zero-day attacks still remained a challenging task. Traditional signature-based Network Intrusion Detection Systems (NIDS) cannot detect such anomalies as there exists no known signature for them. Moreover, Machine Learning-based (ML-based) NIDS trained with a vanilla supervised learning method cannot detect them as they come from a different distribution compared to what the model has been trained on. Domain adaptation techniques help transfer the knowledge gained from a labeled source domain to an unlabeled target domain. Such techniques have the potential to make a model trained on a dataset containing a few network attacks to detect new types of anomalies that might happen in the future. However, recent domain adaptation methods have been mostly designed for images and provide very limited benefits when applied to network traffic. In this paper, we introduce Proportional Progressive Pseudo-Labeling (PPPL), an effective approach for building a more general domain adaptation technique that can be leveraged to detect unseen anomalies in network systems. At the beginning of the training phase, PPPL progressively reduces target domain classification error, by training the model directly with pseudo-labeled target domain samples, while excluding samples with pseudo-labels that are more likely to be wrong from the training set and postponing training on such samples. Our evaluation conducted on the CICIDS2017 dataset shows that PPPL can significantly outperform other baselines in detecting unseen anomalies with up to 58% improvement based on the average F1 score.

Deep learning for detecting logic-flaw-exploiting network attacks: An end-to-end approach

Network attacks have become a major security concern for organizations worldwide. A category of network attacks that exploit the logic (security) flaws of a few widely-deployed authentication protocols has been commonly observed in recent years. Such logic-flaw-exploiting network attacks often do not have distinguishing signatures, and can thus easily evade the typical signature-based network intrusion detection systems. Recently, researchers have applied neural networks to detect network attacks with network logs. However, public network data sets have major drawbacks such as limited data sample variations and unbalanced data with respect to malicious and benign samples. In this paper, we present a new end-to-end approach based on protocol fuzzing to automatically generate high-quality network data, on which deep learning models can be trained for network attack detection. Our findings show that protocol fuzzing can generate data samples that cover real-world data, and deep learning models trained with fuzzed data can successfully detect the logic-flaw-exploiting network attacks.

A Fuzzing-Based Method for Testing Rules in Intrusion Detection Systems in 6G Networks

With the growing development of 6G techniques, it is more challenging than ever to tune rules in the signature-based network intrusion detection system (NIDS). Often, network managers have to tailor the ruleset manually. However, rules are prone to be problematic because of improper rule configuration. Moreover, the 6G network poses new challenges and requirements in rule verification. Hence, to evaluate the soundness of NIDS rules in 6G networks, we propose a new fuzzing solution for testing rules of signature-based NIDS. To achieve stateful fuzzing, we used a tunable responder to generate context-specific responses to construct network states. Moreover, fuzzing strategies (e.g., signature combination, repetition, and obfuscation) tailored for NIDS rules are designed to ensure effectiveness and rule coverage in fuzzing. With our prototype implementation, we find problems in the detecting rules of NIDSs, including conflicted rule, performance degradation, and NIDS bypassing, which seriously threaten the security of 6G networks.

Benchmarking Deep Learning Methods for Behaviour-Based Network Intrusion Detection

Network security encloses a wide set of technologies dealing with intrusions detection. Despite the massive adoption of signature-based network intrusion detection systems (IDSs), they fail in detecting zero-day attacks and previously unseen vulnerabilities exploits. Behaviour-based network IDSs have been seen as a way to overcome signature-based IDS flaws, namely through the implementation of machine-learning-based methods, to tolerate new forms of normal network behaviour, and to identify yet unknown malicious activities. A wide set of machine learning methods has been applied to implement behaviour-based IDSs with promising results on detecting new forms of intrusions and attacks. Innovative machine learning techniques have emerged, namely deep-learning-based techniques, to process unstructured data, speed up the classification process, and improve the overall performance obtained by behaviour-based network intrusion detection systems. The use of realistic datasets of normal and malicious networking activities is crucial to benchmark machine learning models, as they should represent real-world networking scenarios and be based on realistic computers network activity. This paper aims to evaluate CSE-CIC-IDS2018 dataset and benchmark a set of deep-learning-based methods, namely convolutional neural networks (CNN) and long short-term memory (LSTM). Autoencoder and principal component analysis (PCA) methods were also applied to evaluate features reduction in the original dataset and its implications in the overall detection performance. The results revealed the appropriateness of using the CSE-CIC-IDS2018 dataset to benchmark supervised deep learning models. It was also possible to evaluate the robustness of using CNN and LSTM methods to detect unseen normal activity and variations of previously trained attacks. The results reveal that feature reduction methods decreased the processing time without loss of accuracy in the overall detection performance.

The Next-Generation NIDS Platform: Cloud-Based Snort NIDS Using Containers and Big Data

Snort is a well-known, signature-based network intrusion detection system (NIDS). The Snort sensor must be placed within the same physical network, and the defense centers in the typical NIDS architecture offer limited network coverage, especially for remote networks with a restricted bandwidth and network policy. Additionally, the growing number of sensor instances, followed by a quick increase in log data volume, has caused the present system to face big data challenges. This research paper proposes a novel design for a cloud-based Snort NIDS using containers and implementing big data in the defense center to overcome these problems. Our design consists of Docker as the sensor’s platform, Apache Kafka, as the distributed messaging system, and big data technology orchestrated on lambda architecture. We conducted experiments to measure sensor deployment, optimum message delivery from the sensors to the defense center, aggregation speed, and efficiency in the data-processing performance of the defense center. We successfully developed a cloud-based Snort NIDS and found the optimum method for message-delivery from the sensor to the defense center. We also succeeded in developing the dashboard and attack maps to display the attack statistics and visualize the attacks. Our first design is reported to implement the big data architecture, namely, lambda architecture, as the defense center and utilize rapid deployment of Snort NIDS using Docker technology as the network security monitoring platform.

SDN-Based Network Intrusion Detection as DDoS defense system for Virtualization Environment

Nowadays, DDoS attacks are often aimed at cloud computing environments, as more people use virtualization servers. With so many Nodes and distributed services, it will be challenging to rely solely on conventional networks to control and monitor intrusions. We design and deploy DDoS attack defense systems in virtualization environments based on Software-defined Networking (SDN) by combining signature-based Network Intrusion Detection Systems (NIDS) and sampled flow (sFlow). These techniques are practically tested and evaluated on the Proxmox production Virtualization Environment testbed, adding High Availability capabilities to the Controller. The evaluation results show that it promptly detects several types of DDoS attacks and mitigates their negative impact on network performance. Moreover, it also shows good results on Quality of Service (QoS) parameters such as average packet loss about 0 %, average latency about 0.8 ms, and average bitrate about 860 Mbit/s.

BUILDING A DYNAMIC SCALABLE PARALLEL CLOUD-BASED SNORT NIDS USING CONTAINERS AND BIG DATA

Snort is one of the well-known signature-based network intrusion detection systems (NIDS). The Snort sensor placement must be in the same physical network. The defense center in the typical NIDS architecture cause limited network coverage to be monitored, especially for remote networks with restricted bandwidth and network policy. Moreover, the increasing number of sensor instances, followed by a rapid increase in log data volume, caused the existing system to face Big data challenges. This research paper aims to propose a novel design of cloud-based Snort NIDS using containers and implementing Big data in the defense center to overcome these problems. Our design consists of Docker as the sensor's platform, Apache Kafka as the distributed messaging system, and various big data technology orchestrated on lambda architecture. Experiments are conducted to measure sensor deployment, optimum message delivery from sensors to the defense center, and aggregation speed, and data processing performance efficiency on the defense center. In summary, we successfully developed a cloud-based Snort NIDS and found the optimum message delivery method from the sensor to the defense center. Our design also represents the first report on implementing the Big data architecture, namely lambda architecture, to the defense center as a part of a network security monitoring platform.

Variables influencing the effectiveness of signature-based network intrusion detection systems

ABSTRACT Contemporary organizations often employ signature-based network intrusion detection systems to increase the security of their computer networks. The effectiveness of a signature-based system primarily depends on the quality of the rules used to associate system events to known malicious behavior. However, the variables that determine the quality of rulesets is relatively unknown. This paper empirically analyzes the detection probability in a test involving Snort for 1143 exploitation attempts and 12 Snort rulesets created by the Emerging Threats Labs and the Sourcefire Vulnerability Research Team. The default rulesets from Emerging Threats raised priority-1-alerts for 39% of the exploit attempts compared to 31% for rulesets from the Vulnerability Research Team. The following features predict detection probability: if the exploit is publicly known, if the ruleset references the exploited vulnerability, the payload, the type of software targeted, and the operating system of the targeted software. The importance of these variables depends on the ruleset used and whether default rules are used. A logistic regression model with these variables classifies 69–92% of the cases correctly for the different rulesets.

On High-Speed Flow-Based Intrusion Detection Using Snort-Compatible Signatures

Signature-based Network Intrusion Detection Systems (NIDS) have become state-of-the-art in modern network security solutions. However, most systems are not designed for modern high-speed network links. In the field of network monitoring, an alternative solution has become the choice for such high-speed networks. Flow-monitoring, typically based on the Internet Protocol Flow Information Export (IPFIX) standard, now goes well beyond collecting statistical information about network connections. Current solutions are even able to include selected parts of the payload in these Flows to be used in conjunction with NIDS. Recently, we extended this concept to application layer HTTP Flows. We now present our improved version of the IPFIX-based Signature-based Intrusion Detection System (FIXIDS). <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Fixids</small> makes use of HTTP intrusion detection signatures from the popular Snort system and applies them to incoming IPFIX-conforming HTTP Flows. Our evaluation shows that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Fixids</small> can deal with four times higher network data rates without drops compared to Snort, while maintaining the same event detection rate. Furthermore, a substantial part of the data traffic can be outsourced to <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Fixids</small> so that Snort can be relieved of a significant portion of rules and traffic. This increases both the detection rate and the data rate the overall security appliance can handle.

Synthesis of reconfigurable information security hardware on HPC platforms

The main purpose of a signature-based network intrusion detection system (NIDS) is to inspect network packet contents against tens of thousands of predefined malicious patterns. Unlike the firewall, NIDS examines not only packet headers, but also the packet bodies. The multi-pattern string matching task is a specific type of string matching functionality to search an input stream for a set of patterns rather than a single pattern. Due to rising traffic rates, increasing number and sophistication of attacks and the collapse of Moore's law for sequential processing, traditional software solutions can no longer meet the high requirements of today’s security challenges. Therefore, hardware approaches are proposed to accelerate pattern matching. Combining the flexibility of software and the nearASIC performance, reconfigurable FPGA-based devices have become increasingly popular for this purpose. Unfortunately, the development of complex reconfigurable devices is a very difficult craft. Users of NIDS which are usually system administrators have not neither enough qualification, nor computing resources to fulfill such a work. On the other hand specificities of security tasks require frequent execution of dynamic re-synthesis of reconfigurable accelerators. To solve this problem, a centralized system based on GRID and Cloud platforms was proposed. Such approach moves design and computation complexities from LANs to HPC. An experimental system was constructed and tested. First results are received and discussed. Preliminary comparison of GRID and Cloud technologies is made. Besides cybersecurity, high-speed multi-pattern matching is required for such important applications as data mining, XML switching, QoS management, VoIP filtering, cache replication etc.

Towards effective and robust list-based packet filter for signature-based network intrusion detection: an engineering approach

Network intrusion detection systems (NIDSs) which aim to identify various attacks, have become an essential part of current security infrastructure. In particular, signature-based NIDSs are being w...

Length-Bounded Hybrid CPU/GPU Pattern Matching Algorithm for Deep Packet Inspection

Since frequent communication between applications takes place in high speed networks, deep packet inspection (DPI) plays an important role in the network application awareness. The signature-based network intrusion detection system (NIDS) contains a DPI technique that examines the incoming packet payloads by employing a pattern matching algorithm that dominates the overall inspection performance. Existing studies focused on implementing efficient pattern matching algorithms by parallel programming on software platforms because of the advantages of lower cost and higher scalability. Either the central processing unit (CPU) or the graphic processing unit (GPU) were involved. Our studies focused on designing a pattern matching algorithm based on the cooperation between both CPU and GPU. In this paper, we present an enhanced design for our previous work, a length-bounded hybrid CPU/GPU pattern matching algorithm (LHPMA). In the preliminary experiment, the performance and comparison with the previous work are displayed, and the experimental results show that the LHPMA can achieve not only effective CPU/GPU cooperation but also higher throughput than the previous method.

A Hybrid CPU/GPU Pattern-Matching Algorithm for Deep Packet Inspection.

The large quantities of data now being transferred via high-speed networks have made deep packet inspection indispensable for security purposes. Scalable and low-cost signature-based network intrusion detection systems have been developed for deep packet inspection for various software platforms. Traditional approaches that only involve central processing units (CPUs) are now considered inadequate in terms of inspection speed. Graphic processing units (GPUs) have superior parallel processing power, but transmission bottlenecks can reduce optimal GPU efficiency. In this paper we describe our proposal for a hybrid CPU/GPU pattern-matching algorithm (HPMA) that divides and distributes the packet-inspecting workload between a CPU and GPU. All packets are initially inspected by the CPU and filtered using a simple pre-filtering algorithm, and packets that might contain malicious content are sent to the GPU for further inspection. Test results indicate that in terms of random payload traffic, the matching speed of our proposed algorithm was 3.4 times and 2.7 times faster than those of the AC-CPU and AC-GPU algorithms, respectively. Further, HPMA achieved higher energy efficiency than the other tested algorithms.

Enhancing the performance of signature-based network intrusion detection systems: an engineering approach

Signature-based network intrusion detection systems (NIDSs) have been popularly implemented in different organisations, with the purpose of defending against various attacks. However, it is identified that these systems suffer from three major issues in practical applications such as overload packets, expensive signature matching and massive false alarms, which would significantly decrease the effectiveness of these systems. In this paper, an adaptive framework is proposed to improve the overall performance of a signature-based NIDS such as Snort regarding the aforementioned issues. This framework is further implemented in an engineering way, in which a trust-based packet filter with an exclusive signature matching scheme, and an intelligent machine learning-based false alarm filter aiming to reduce target packets, improve the process of signature matching and decrease the number of false alarms are constructed, respectively. In the evaluation, the experimental results on a well-known benchmark and a real...

EFM: Enhancing the performance of signature-based network intrusion detection systems using enhanced filter mechanism

Signature-based network intrusion detection systems (NIDSs) have been widely deployed in current network security infrastructure. However, these detection systems suffer from some limitations such as network packet overload, expensive signature matching and massive false alarms in a large-scale network environment. In this paper, we aim to develop an enhanced filter mechanism (named EFM) to comprehensively mitigate these issues, which consists of three major components: a context-aware blacklist-based packet filter, an exclusive signature matching component and a KNN-based false alarm filter. The experiments, which were conducted with two data sets and in a network environment, demonstrate that our proposed EFM can overall enhance the performance of a signature-based NIDS such as Snort in the aspects of packet filtration, signature matching improvement and false alarm reduction without affecting network security.

Towards adaptive character frequency-based exclusive signature matching scheme and its applications in distributed intrusion detection

Network intrusion detection systems (NIDSs), especially signature-based NIDSs, are being widely deployed in a distributed network environment with the purpose of defending against a variety of network attacks. However, signature matching is a key limiting factor to limit and lower the performance of a signature-based NIDS in a large-scale network environment, in which the cost is at least linear to the size of an input string. The overhead network packets can greatly reduce the effectiveness of such detection systems and heavily consume computer resources. To mitigate this issue, a more efficient signature matching algorithm is desirable. In this paper, we therefore develop an adaptive character frequency-based exclusive signature matching scheme (named ACF-EX) that can improve the process of signature matching for a signature-based NIDS. In the experiment, we implemented the ACF-EX scheme in a distributed network environment, evaluated it by comparing with the performance of Snort. In addition, we further apply this scheme to constructing a packet filter that can filter out network packets by conducting exclusive signature matching for a signature-based NIDS, which can avoid implementation issues and improve the flexibility of the scheme. The experimental results demonstrate that, in the distributed network environment, the proposed ACF-EX scheme can positively reduce the time consumption of signature matching and that our scheme is promising in constructing a packet filter to reduce the burden of a signature-based NIDS.

Human perspective to anomaly detection for cybersecurity

Traditionally signature-based network Intrusion Detection Systems (IDS) rely on inputs from domain experts and can only identify the attacks if they occur as individual event. IDS generate large number of alerts and it becomes very difficult for human users to go through each message. Previous researches have proposed analytics based approaches to analyze IDS alert patterns based on anomaly detection models, multi-steps models or probabilistic approaches. However, due to the complexities of network intrusions, it is impossible to develop all possible attack patterns or to avoid false positives. With the advance in technologies and popularity of networks in our daily life, it is becoming more and more difficult to detect network intrusions. However, no matter how rapid the technologies change, the human behaviors behind the cyber attacks stay relatively constant. This provides us an opportunity to develop an improved system to detect the unusual cyber attacks. In this paper, we developed four network intrusion models based on consideration of human factors. We then tested these models on ITOC Cyber Defense Competition (CDX) 2009 data. Our results are encouraging. These Models are not only able to recognize most network attacks identified by SNORT log alerts, they are also able to distinguish the non-attack network traffic that was potentially missed by SNORT as indicated by ground truth validation of the data.

Analysis of update delays in signature-based network intrusion detection systems

Network Intrusion Detection Systems (NIDS) play a fundamental role on security policy deployment and help organizations in protecting their assets from network attacks. Signature-based NIDS rely on a set of known patterns to match malicious traffic. Accordingly, they are unable to detect a specific attack until a specific signature for the corresponding vulnerability is created, tested, released and deployed. Although vital, the delay in the updating process of these systems has not been studied in depth. This paper presents a comprehensive statistical analysis of this delay in relation to the vulnerability disclosure time, the updates of vulnerability detection systems (VDS), the software patching releases and the publication of exploits. The widely deployed NIDS Snort and its detection signatures release dates have been used. Results show that signature updates are typically available later than software patching releases. Moreover, Snort rules are generally released within the first 100 days from the vulnerability disclosure and most of the times exploits and the corresponding NIDS rules are published with little difference. Implications of these results are drawn in the context of security policy definition. This study can be easily kept up to date due to the methodology used.

A fast pattern matching algorithm with multi-byte search unit for high-speed network security

A signature-based intrusion detection system identifies intrusions by comparing the data traffic with known signature patterns. In this process, matching of packet strings against signature patterns is the most time-consuming step and dominates the overall system performance. Many signature-based network intrusion detection systems (NIDS), e.g., the Snort, employ one or multiple pattern matching algorithms to detect multiple attack types. So far, many pattern matching algorithms have been proposed. Most of them use single-byte standard unit for search, while a few algorithms such as the Modified Wu–Manber (MWM) algorithm use typically two-byte unit, which guarantees better performance than others even as the number of different signatures increases. Among those algorithms, the MWM algorithm has been known as the fastest pattern matching algorithm when the patterns in a rule set rarely appear in packets. However, the matching time of the MWM algorithm increases as the length of the shortest pattern in a signature group decreases. In this paper, by extending the length of the shortest pattern, we minimize the pattern matching time of the algorithm which uses multi-byte unit. We propose a new pattern matching algorithm called the L +1-MWM algorithm for multi-pattern matching. The proposed algorithm minimizes the performance degradation that is originated from the dependency on the length of the shortest pattern. We show that the L +1-MWM algorithm improves the performance of the MWM algorithm by as much as 20% in average under various lengths of shortest patterns and normal traffic conditions. Moreover, when the length of the shortest pattern in a rule set is less than 5, the L +1-MWM algorithm shows 38.87% enhancement in average. We also conduct experiments on a real campus network and show that 12.48% enhancement is obtained in average. In addition, it is shown that the L +1-MWM algorithm provides a better performance than the MWM algorithm by as much as 25% in average under various numbers of signatures and normal traffic conditions, and 20.12% enhancement in average with real on-line traffic.