Snort Rule Set Research Articles

FPGA-CPU Architecture Accelerated Regular Expression Matching With Fast Preprocessing

Abstract Regular Expression Matching (REM) is the core of Deep Packet Inspection (DPI), which is important for various network security applications. The burgeoning Software Defined Network and Network Function Virtualization technologies make the network evolve more dynamic, which brings serious challenges for DPI engines to achieve high matching performance with fast rule-set update capability. To meet these challenges, this paper proposes a heterogeneous Field Programmable Gate Array (FPGA)-Central Processing Unit (CPU) architecture to accelerate Deterministic Finite Automaton (DFA)-based REM with high preprocessing performance. Firstly, a novel regex decomposition technique is proposed to solve the DFA state explosion problem, which splits each regex into one prefix and several postfixes. Secondly, heterogeneous architecture is presented to collaboratively handle regex matching, in which prefixes are matched in parallel in an FPGA and postfixes are matched in a CPU. To further improve the matching performance, several well-designed DFA compression techniques and regex decomposition optimizations are proposed. Our design has been implemented in a DPI prototype employing a medium-end FPGA. Extensive experiments are conducted to evaluate the performance. Results reveal that our proposed architecture achieves 6.33 Gbps matching throughput on the Snort rule-set (v3.0), which is close to state-of-the-art FPGA NFA-based schemes. However, the rule-set preprocessing time is significantly reduced to <7 minutes, compared with up to several hours of FPGA NFA-based countermeasures.

Bridging the Last-Mile Gap in Network Security via Generating Intrusion-Specific Detection Patterns through Machine Learning

With successful machine learning applications in many fields, researchers tried to introduce machine learning into intrusion detection systems for building classification models. Although experimental results showed that these classification models could produce higher accuracy in predicting network attacks on the offline datasets, compared with the operational intrusion detection systems, machine learning is rarely deployed in the real intrusion detection environment. This is what we call the last mile problem with the machine learning approach to network intrusion detection, the discrepancy between the strength and requirements of machine learning and network operational semantics. In this paper, we aim to bridge the aforementioned gap. In particular, an LCC-RF-RFEX feature selection approach is proposed to select optimal features of the specific type of attacks from dataset, and then, an intrusion-specific approach is introduced to convert them into detection patterns that can be used by the nonmachine-learning detector for the corresponding specific attack detection in the real-world network environment. To substantiate our approach, we take Snort, KDDCup’99 dataset, and Dos attacks as the experimental subjects to demonstrate how to close the last-mile gap. For the specific type of Dos attacks in the KDDCup’99 dataset, we use the LCC-RF-RFEX method to select optimal feature subset and utilize our intrusion-specific approach to generate new rules in Snort by using them. Comparing performance differences between the existing Snort rule set and our augmented Snort rule set with regard to Dos attacks, the experimental results showed that our approach expanded Snort’s detection capability of Dos attacks, on average, reduced up to 25.28% false-positive alerts for Teardrop attacks and Synflood attacks, and decreased up to 98.87% excessive alerts for Mail bomb attacks.

Variables influencing the effectiveness of signature-based network intrusion detection systems

ABSTRACT Contemporary organizations often employ signature-based network intrusion detection systems to increase the security of their computer networks. The effectiveness of a signature-based system primarily depends on the quality of the rules used to associate system events to known malicious behavior. However, the variables that determine the quality of rulesets is relatively unknown. This paper empirically analyzes the detection probability in a test involving Snort for 1143 exploitation attempts and 12 Snort rulesets created by the Emerging Threats Labs and the Sourcefire Vulnerability Research Team. The default rulesets from Emerging Threats raised priority-1-alerts for 39% of the exploit attempts compared to 31% for rulesets from the Vulnerability Research Team. The following features predict detection probability: if the exploit is publicly known, if the ruleset references the exploited vulnerability, the payload, the type of software targeted, and the operating system of the targeted software. The importance of these variables depends on the ruleset used and whether default rules are used. A logistic regression model with these variables classifies 69–92% of the cases correctly for the different rulesets.

Deep learning-based feature extraction and optimizing pattern matching for intrusion detection using finite state machine

Deep learning has gained popularity for feature extraction in the field of Network Intrusion Detection and Prevention System (NIDPS) to extract the patterns matching and secure the networks by detecting the unknown and malicious activities. The malicious activities and security attacks are disturbing the normal operations of networks. The new attacks are difficult to monitor due to their new features and pattern types. Different types of methods have been adopted for feature extraction and pattern matching. Deep learning is one of them and subfield of machine learning where it solves the optimization issues layer-wise by looking at the deep structure. On the other hand, pattern matching is another considerable method for intrusion detection due to its variety of applications. However, pattern matching methods are consuming more than 70% of the total running time and cause overhead. In this paper, we propose two methods including Deep Learning-based Feature Extraction (DLFE) and Optimization of Pattern Matching (OPM) for NIDPS systems to optimizes the pattern matching engine in intrusion detection. The experiments are performed by using the snort ruleset for pattern matching and obtained the results. The experiment results show the better performance of proposed methods in terms of time, throughput, and memory.

Memristor TCAMs Accelerate Regular Expression Matching for Network Intrusion Detection

We propose memristor-based TCAMs (Ternary Content Addressable Memory) circuits to accelerate Regular Expression (RegEx) matching through in memory processing of finite automata. RegEx matching is a key function in network security to find malicious actors. However, RegEx matching latency and power can be incredibly high and current proposals are challenged to perform wire-speed matching for large rulesets. Our approach dramatically decreases operating power, enables high throughput, and the use of nanoscale memristor TCAM circuits (mTCAMs) enables compression techniques to expand rulesets. We fabricated and demonstrated nanoscale memristor TCAM cells. SPICE simulations investigate performance at scale and a mTCAM dynamic power model using 16 nm layout parameters demonstrates ~0.2 fJ/bit/search energy for a 36 × 250 mTCAM array. A tiled architecture is proposed to implement a Snort ruleset and assess application performance. Compared to a state-of-the-art FPGA approach (2 Gbps, ~1 W), we show ×4 throughput (8 Gbps) at 55% the power (0.55 W) without standard TCAM power-saving techniques. Our performance comparison improves further when striding (searching multiple characters at once) is considered, resulting in 47.2 Gbps at 1.2 W for our approach compared to 3.9 Gbps at 630 mW for strided FPGA NFA, demonstrating a promising path to wire-speed RegEx matching on large scale rulesets.

A Statistical and Cluster Analysis Exploratory Study of Snort Rules

While many studies applying machine learning algorithms to signature based intrusion detection systems (IDSs) have focused on log analysis, another avenue that might have the potential to yield additional insight into the intrusion detection problem is analysis of the rules that lie at the heart of an IDS. A signature-based IDS is generally ineffective against zero-day attacks; however, an analysis of the types of attack signatures encountered by such a system over its history, could provide guidance about future attacks. This history is essentially encapsulated into the rules of the IDS itself. An examination of rules can reveal a variety of useful information about the kinds of traffic that a network considers to be malicious. This research performs a statistical and machine learning cluster analysis of Snort rules, with a focus on the network protocols used by the rules. It proceeds in two phases. In Phase 1, algorithms are developed to extract protocol information from Snort rules and to determine their distribution across rule sets. This component of the research shows that there are three major types of protocols used in Snort rules: Transmission Control Protocol (TCP), User Datagram Protocol (UDP) and Internet Control Message Protocol (ICMP). It also provides the frequency (or cardinality) of each such protocol per rule set. Phase 2, which focused on the default enabled rules of the latest Snort rule version, performs cluster analyses using the following three approaches: k-means algorithm, a hierarchical agglomerative clustering algorithm, and a density based clustering algorithm. This component of the research illustrates that with respect to the number of protocols per rule set, TCP use is dominant and that it is reasonable to divide Snort rule sets into three principal clusters. One cluster consists of a single rule set and is characterized by a preponderance of UDP protocol usage among its rules. A second cluster consists of four rule sets and is distinguished by each rule set containing more than 900 TCP-based rules. The last cluster consists of all other rulesets. TCP based rules for rule sets in this cluster are all below 500. Comparison among the three clustering algorithms using the silhouette metric, demonstrate them to be very effective, with negligible variation in performance.

Hybrid memory-efficient multimatch packet classification for NIDS

Network applications such as network intrusion detection systems (NIDSs) require multimatch packet classification, where all matched results need to be reported. Most researchers have adopted a TCAM-based architecture to enhance system performance, but TCAM consumes high amounts of power and requires a lot of memory resources. In this paper, we analyze the characteristics of the Snort rule set, and propose an memory-efficient multimatch packet classification architecture for NIDS using the result of analysis. The proposed hybrid architecture uses hash-based matching for searching single port numbers and k-ary tree matching for searching range port numbers and is synthesized on Altera Stratix IV FPGA. Compared with previous TCAM-based architectures, our design achieves over four times improvement in memory requirement and power consumption. Our architecture sustains 16.8–67.4Gbps throughput for minimum size (40bytes) packets.

A DFA with Extended Character-Set for Fast Deep Packet Inspection

Deep packet inspection (DPI), based on regular expressions, is expressive, compact, and efficient in specifying attack signatures. We focus on their implementations based on general-purpose processors that are cost-effective and flexible to update. In this paper, we propose a novel solution, called deterministic finite automata with extended character-set (DFA/EC), which can significantly decrease the number of states through doubling the size of the character-set. Unlike existing state reduction algorithms, our solution requires only a single main memory access for each byte in the traffic payload, which is the minimum. We perform experiments with several Snort rule-sets. Results show that, compared to DFAs, DFA/ECs are very compact and are over four orders of magnitude smaller in the best cases; DFA/ECs also have smaller memory bandwidth and run faster. We believe that DFA/EC will lay a groundwork for a new type of state compression technique in fast packet inspection.

A distributed simulation framework for modeling cyber attacks and the evaluation of security measures

The aim of this work is to propose a framework for the distributed simulation of cyber attacks based on high-level architecture (HLA), which is a commonly used standard for distributed simulations. The proposed framework and the corresponding simulator, which is called the distributed cyber attack simulator (abbreviated by DCAS), help administrators to model and evaluate the security measures of the networks. At the core of the DCAS is a simulation engine based on Portico, which is an open source HLA run-time infrastructure. The DCAS works in two modes: interactive and automated. Three types of simulation components (which are called federates in HLA terminology) are considered in the framework: the (1) network federate, (2) attacker federate and (3) defender federate. The simulator provides features for graphical design of the network models, animated traffic simulation, data collection, statistical analysis and different consoles for attacking and defending elements (e.g., intrusion detection systems, intrusion prevention systems). To increase the fidelity of the simulation outputs, real-world payloads are used by the DCAS. All the exploits information and the parameters of various network elements are automatically extracted from the open source vulnerability database. Also, the Snort rule-set is used as the signature database of the defending elements. The architecture and algorithms of the DCAS and the corresponding underlying simulation engine plus the security evaluation results of two illustrative examples are presented in this paper.

High Performance Network Security Using NIDS Approach

Ever increasing demand of good quality communication relies heavily on Network Intrusion Detection System (NIDS). Intrusion detection for network security demands high performance. This paper gives a description of the available approaches for a network intrusion detection system in both software and hardware implementation. This paper gives a description of the structure of Snort rule set which is a very popular software signature and anomaly based Intrusion Detection and prevention system. This paper also discusses the merit of FPGA devices to be used in network intrusion detection system implementation and the approaches used in hardware implementation of NIDS.

A quality metric for IDS signatures: in the wild the size matters

The manual forensics investigation of security incidents is an opaque process that involves the collection and correlation of diverse evidence. In this work we first conduct a complex experiment to expand our understanding of forensics analysis processes. During a period of 4 weeks, we systematically investigated 200 detected security incidents about compromised hosts within a large operational network. We used data from four commonly used security sources, namely Snort alerts, reconnaissance and vulnerability scanners, blacklists, and a search engine, to manually investigate these incidents. Based on our experiment, we first evaluate the (complementary) utility of the four security data sources and surprisingly find that the search engine provided useful evidence for diagnosing many more incidents than more traditional security sources, i.e., blacklists, reconnaissance, and vulnerability reports. Based on our validation, we then identify and make publicly available a list of 165 good Snort signatures, i.e., signatures that were effective in identifying validated malware without producing false positives. In addition, we analyze the characteristics of good signatures and identify strong correlations between different signature features and their effectiveness, i.e., the number of validated incidents in which a good signature is identified. Based on our experiment, we finally introduce an IDS signature quality metric that can be exploited by security specialists to evaluate the available rulesets, prioritize the generated alerts, and facilitate the forensics analysis processes. We apply our metric to characterize the most popular Snort rulesets. Our analysis of signatures is useful not only for configuring Snort but also for establishing best practices and for teaching how to write new IDS signatures.

A Studying on Implementation of NIDS Pattern Matching Based on FPGA

Intrusion detection for network security is an application area demanding high throughput. The pattern matching in intrusion detection requires extremely high performance to process string matching. Most of pattern matching using software has many time complexities and cannot reach the requirements of high throughput. The pattern matching using hardware considerably improves the speed of matching and has several other advantages. This paper describes a FPGA-based pattern matching architecture, using hashing method called XOR Hashing. The proposed method updates new patterns without reconfiguration and processes the collision and has high matching performance. The proposed system implements the pattern matching by using Snort rule-set, an open source Network Intrusion Detection and has simulation processing on PC. Compared with existing hardware method, the results explained that our method has relatively high performance for the pattern matching and can else process the pattern matching with high performance on low–cost FPGA device.

Efficient Pattern Matching Algorithm for Memory Architecture

Network intrusion detection system is used to inspect packet contents against thousands of predefined malicious or suspicious patterns. Because traditional software alone pattern matching approaches can no longer meet the high throughput of today's networking, many hardware approaches are proposed to accelerate pattern matching. Among hardware approaches, memory-based architecture has attracted a lot of attention because of its easy reconfigurability and scalability. In order to accommodate the increasing number of attack patterns and meet the throughput requirement of networks, a successful network intrusion detection system must have a memory-efficient pattern-matching algorithm and hardware design. In this paper, we propose a memory-efficient pattern-matching algorithm which can significantly reduce the memory requirement. For Snort rule sets, the new algorithm achieves 21% of memory reduction compared with the traditional Aho-Corasick algorithm. In addition, we can gain 24% of memory reduction by integrating our approach to the bit-split algorithm which is the state-of-the-art memory-based approach.

PAMELA: Pattern Matching Engine with Limited-Time Update for NIDS/NIPS

Several hardware-based pattern matching engines for network intrusion/prevention detection systems (NIDS/NIPSs) can achieve high throughput with less hardware resources. However, their flexibility to update new patterns is limited and still challenging. This paper describes a PAttern Matching Engine with Limited-time updAte (PAMELA) engine using a recently proposed hashing algorithm called Cuckoo Hashing. PAMELA features on-the-fly pattern updates without reconfiguration, more efficient hardware utilization, and higher performance compared with other works. First, we implement the improved parallel exact pattern matching with arbitrary length based on Cuckoo Hashing and linked-list technique. Second, while PAMELA is being updated with new attack patterns, both stack and FIFO are utilized to bound insertion time due to the drawback of Cuckoo Hashing and to avoid interruption of input data stream. Third, we extend the system for multi-character processing to achieve higher throughput. Our engine can accommodate the latest Snort rule-set, an open source NIDS/NIPS, and achieve the throughput up to 8.8 Gigabit per second while consuming the lowest amount of hardware. Compared to other approaches, ours is far more efficient than any other implemented on Xilinx FPGA architectures.

FPGA-Based Intrusion Detection System for 10 Gigabit Ethernet

The present paper describes an implementation of an intrusion detection system (IDS) on an FPGA for 10 Gigabit Ethernet. The system includes an exact string matching circuit for 1,225 Snort rules on a single device. A number of studies have examined string matching circuits for IDS. However, implementing a circuit that processes a large rule set at high throughput is difficult. In a previous study, we proposed a method for generating an NFA-based string matching circuit that has expandability of processing data width and drastically reduced resource requirements. In the present paper, we implement an IDS circuit that processes 1,225 Snort rules at 10 Gbps with a single Xilinx Virtex-II Pro xc2vp-100 using the NFA-based method. The proposed circuit also provides packet filtering for an intrusion protection system (IPS). In addition, we developed a tool for automatically generating the Verilog HDL source code of the IDS circuit from a Snort rule set. Using the FPGA and the IDS circuit generator, the proposed system is able to update the matching rules corresponding to new intrusions and attacks. We implemented the IDS circuit on an FPGA board and evaluated its accuracy and throughput. As a result, we confirmed in a test that the circuit detects attacks perfectly at the wire speed of 10 Gigabit Ethernet.

A fast string-matching algorithm for network processor-based intrusion detection system

Network intrusion detection systems (NIDSs) are one of the latest developments in security. The matching of packet strings against collected signatures dominates signature-based NIDS performance. Network processors are also one of the fastest growing segments of the semiconductor market, because they are designed to provide scalable and flexible solutions that can accommodate change quickly and economically. This work presents a fast string-matching algorithm (called FNP) over the network processor platform that conducts matching sets of patterns in parallel. This design also supports numerous practical features such as case-sensitive string matching, signature prioritization, and multiple-content signatures. This efficient multiple-pattern matching algorithm utilizes the hardware facilities provided by typical network processors instead of employing the external lookup co-processors. To verify the efficiency and practicability of the proposed algorithm, it was implemented on the Vitesse IQ2000 network processor platform. The searching patterns used in the present experiments are derived from the well-known Snort ruleset cited by most open-source and commercial NIDSs. This work shows that combining our string-matching methodology, hashing engine supported by most network processors, and characteristics of current Snort signatures frequently improves performance and reduces number of memory accesses compared to conventional string-matching algorithms. Another contribution of this work is to highlight that, besides total number of searching patterns, shortest pattern length is also a major influence on NIDS multipattern matching algorithm performance.