High-speed Network Intrusion Detection Research Articles

Hybrid Classification for High-Speed and High-Accuracy Network Intrusion Detection System

Cybercrime is growing at a rapid pace, and its techniques are becoming more sophisticated. In order to actively cope with such threats, new approaches based on machine learning and requiring less administrator intervention have been proposed, but there are still many technical difficulties in detecting security attacks in real time. To solve this problem, we propose a new machine learning-based real-time intrusion detection algorithm. Unlike the existing approaches, the one proposed can detect the presence of an attack every time a packet is received, enabling real-time detection. In addition, our algorithm effectively reduces the system load, which may significantly increase from real-time detection, compared to non-real-time detection. In the algorithm, the increase in the number of memory accesses can be minimized (to below 30 %) compared to conventional methods. Since the proposed method is pure software-based approach, it has excellent scalability and flexibility against various attacks. Therefore, the proposed method cannot support the high classification performance of the hardware-based method but also the high flexibility of the software-based method simultaneously, it can effectively detect and prevent various cyber-attacks.

Abstracting Packet Header Information for Intrusion Detection in High-Speed Networks

Increase in network traffic coupled with increasing adoption of end-to-end encryption of network packets are two major factors threatening the potency, or even the relevance, of packet-based intrusion detection techniques. Also, end-to-end encryption makes it nearly impossible for network and host-based intrusion detection system to analyze traffic for potential threats and intrusion, hence, the need for an alternative approach. Flow-based intrusion detection system has been proposed as an alternative to a packet-based intrusion detection system as it relies on information embedded in packet header and various statistical analyses of network flow for detecting intrusion. This paper proposes packet header information abstraction model for intrusion detection on the UNSW-NB15 intrusion dataset. Four existing classification algorithms which include: Classification and Regression Tree (CART), Naïve Bayes (NB), K-Nearest Neighbour (KNN), and Support Vector Machine (SVM) are used to evaluate the degree of representativeness of the proposed model using accuracy, sensitivity and specificity evaluation metrics. An average accuracy of 97.95% was recorded across the four models with the minimum accuracy of 97.76 on SVM and best accuracy of 98.05% on CART while Sensitivity of 1.0 on both CART and NB shows that the model performs well in correctly identifying attacks in the network. The average specificity of 0.98 is also an indication of low false positive. Results obtained show that the proposed abstraction model achieves high accuracy, sensitivity and specificity. The model can be used as filter on a high-speed network whereby packets flagged as an attack can be subjected to further analysis.Keywords—Data Abstraction, Data Mining,Flow-based, Intrusion detection, Network Security

Approximate reduction of finite automata for high-speed network intrusion detection

We consider the problem of approximate reduction of non-deterministic automata that appear in hardware-accelerated network intrusion detection systems (NIDSes). We define an error distance of a reduced automaton from the original one as the probability of packets being incorrectly classified by the reduced automaton (wrt the probabilistic distribution of packets in the network traffic). We use this notion to design an approximate reduction procedure that achieves a great size reduction (much beyond the state-of-the-art language-preserving techniques) with a controlled and small error. We have implemented our approach and evaluated it on use cases from Snort, a popular NIDS. Our results provide experimental evidence that the method can be highly efficient in practice, allowing NIDSes to follow the rapid growth in the speed of networks.

BigFlow: Real-time and reliable anomaly-based intrusion detection for high-speed networks

Existing machine learning solutions for network-based intrusion detection cannot maintain their reliability over time when facing high-speed networks and evolving attacks. In this paper, we propose BigFlow, an approach capable of processing evolving network traffic while being scalable to large packet rates. BigFlow employs a verification method that checks if the classifier outcome is valid in order to provide reliability. If a suspicious packet is found, an expert may help BigFlow to incrementally change the classification model. Experiments with BigFlow, over a network traffic dataset spanning a full year, demonstrate that it can maintain high accuracy over time. It requires as little as 4% of storage and between 0.05% and 4% of training time, compared with other approaches. BigFlow is scalable, coping with a 10-Gbps network bandwidth in a 40-core cluster commodity hardware.

High Speed and Low Power Architecture for Network Intrusion Detection System

The tremendous growth in the field of modern communication and network systems places demands on the security. As the network complexity grows, the need for the automated detection and timely alert is required to detect the abnormal activities in the network. To diagnose the system against the malicious signatures, a high speed Network Intrusion Detection System is required against the attacks. In the network security applications, Bloom Filters are the key building block. The packets from the high speed link can be easily processed by Bloom Filter using state- of-art hardware based systems. As Bloom Filter and its variant Counting Bloom Filter suffer from False Positive Rate, Multi Hash Counting Bloom Filter architecture is proposed. The proposed work, constitute parallel signature detection improves the False Positive Rate, but the throughput and hardware complexity suffer. To resolve this, a Multi-Level Ranking Scheme is introduced which deduces the 13% - 16% of the power and increases the throughput to 23% - 30%. This work is best suited for signature detection in high speed network.

Constructing important features from massive network traffic for lightweight intrusion detection

Efficiently processing massive data is a big issue in high-speed network intrusion detection, as network traffic has become increasingly large and complex. In this work, instead of constructing a large number of features from massive network traffic, the authors aim to select the most important features and use them to detect intrusions in a fast and effective manner. The authors first employed several techniques, that is, information gain (IG), wrapper with Bayesian networks (BN) and Decision trees (C4.5), to select important subsets of features for network intrusion detection based on KDD'99 data. The authors then validate the feature selection schemes in a real network test bed to detect distributed denial-of-service attacks. The feature selection schemes are extensively evaluated based on the two data sets. The empirical results demonstrate that with only the most important 10 features selected from all the original 41 features, the attack detection accuracy almost remains the same or even becomes better based on both BN and C4.5 classifiers. Constructing fewer features can also improve the efficiency of network intrusion detection.

Algorithms to speedup pattern matching for network intrusion detection systems

High-speed network intrusion detection systems (NIDSes) commonly employ TCAMs for fast pattern matching, and parallel TCAM-based pattern matching algorithms have proven promising to achieve even higher line rate. However, two challenges impede parallel TCAM-based pattern matching engines from being truly scalable, namely: (1) how to implement fine-grained parallelism to optimize load balancing and maximize throughput, and (2) how to reconcile between the performance gain and increased power consumption both due to parallelism. In this paper, we propose two techniques to answer the above challenges yielding an ultra-scalable NIDS. We first introduce the concept of negative pattern matching, by which we can splice flows into segments for fine-grained load balancing and optimized parallel speedup while ensuring correctness. negative pattern matching (NPM) also dramatically reduces the number of Ternary Content Addressable Memory (TCAM) lookups thus reducing the power consumption. Then we propose the idea of exclusive pattern matching, which divides the rule sets into subsets; each subset is queried selectively and independently given a certain input without affecting correctness. In concert, these two techniques improve both the pattern matching throughput and scalability in any scenario. Our experimental results show that up to 90% TCAM lookups can be saved, at the cost of merely 10% additional 2-byte index table lookups in the SRAM.

MULTI-GIGABIT PATTERN FOR DATA IN NETWORK SECURITY

In the current scenario network security is emerging the world. Matching large sets of patterns against an incoming stream of data is a fundamental task in several fields such as network security or computational biology. High-speed network intrusion detection systems (IDS) rely on efficient pattern matching techniques to analyze the packet payload and make decisions on the significance of the packet body. However, matching the streaming payload bytes against thousands of patterns at multi-gigabit rates is computationally intensive. Various techniques have been proposed in past but the performance of the system is reducing because of multi-gigabit rates.Pattern matching is a significant issue in intrusion detection systems, but by no means the only one. Handling multi-content rules, reordering, and reassembling incoming packets are also significant for system performance. We present two pattern matching techniques to compare incoming packets against intrusion detection search patterns. The first approach, decoded partial CAM (DpCAM), pre-decodes incoming characters, aligns the decoded data, and performs logical AND on them to produce the match signal for each pattern. The second approach, perfect hashing memory (PHmem), uses perfect hashing to determine a unique memory location that contains the search pattern and a comparison between incoming data and memory output to determine the match. The suggested methods have implemented in vhdl coding and we use Xilinx for synthesis.

Packet Sampling Applications in Campus High-Speed Network Intrusion Detection System

A technique of high-speed network intrusion detection system based on packet sampling theory is proposed. Starting with basic principles of packet sampling, this paper first analyses the significant mathematical conclusion of sampling strategies, then after discussing current strategies, mechanism and performance of different packet sampling methods, we specify an efficient strategy of packet sampling. Results show that this method can attain above 55% accurate rate with below 1% false rate in 94 specified attacking cases from DARPA 2000 IDS evaluation dataset.

Re-examining the performance bottleneck in a NIDS with detailed profiling

Designing a high-speed network intrusion detection system (NIDS) has attracted much attention in recent years due to ever-increasing amount of network traffic and ever-complicated attacks. Numerous studies have been focusing on accelerating pattern matching for a high-speed design because some early studies observed that pattern matching is a performance bottleneck. However, the effectiveness of such acceleration has been challenged recently. This work therefore re-examines the performance bottleneck by profiling two popular NIDSs, Snort and Bro, with various types of network traffic in detail. In the profiling, we find pattern matching can be dominant in the Snort execution if the entire packet payloads in the connections are scanned, while executing the policy scripts is an obvious bottleneck in the Bro execution. This work suggests three promising directions towards a high-speed NIDS design for future research: a method to precisely specify the possible locations of the signatures in long connections, a compiler to transform the policy scripts to efficient binary codes for execution, and an efficient design of connection tracking and packet reassembly.

Application of unbalanced data approach inhigh-speed network intrusion detection

In view of the current problems of high-speed network intrusion detection system,such as high packet loss rate,slow pace of testing for attacks and unbalanced data for detection,this paper proposed a new two-stage strategy with load balancing intrusion detection model.In the on-line phase,the system captured the packets from network and split into small ones according to the protocol type,and then detected through each sensor.In the off-line phase,training dataset was used to build module which can detect intrusion.The authors discussed different approaches to unbalanced data,empirically evaluated the SMOTE over-sampling approaches and classified with AdaBoost and random forests algorithm.The experimental results show that SMOTE and the AdaBoost Algorithm by using random forests as weak learner not only can provide better performance to small class,but also has steady model building time.

An Active Splitter Architecture for Intrusion Detection and Prevention

State-of-the-art high-speed network intrusion detection and prevention systems are often designed using multiple intrusion detection sensors operating in parallel coupled with a suitable front-end load-balancing traffic splitter. In this paper, we argue that, rather than just passively providing generic load distribution, traffic splitters should implement more active operations on the traffic stream, with the goal of reducing the load on the sensors. We present an active splitter architecture and three methods for improving performance. The first is early filtering/forwarding, where a fraction of the packets is processed on the splitter instead of the sensors. The second is the use of locality buffering, where the splitter reorders packets in a way that improves memory access locality on the sensors. The third is the use of cumulative acknowledgments, a method that optimizes the coordination between the traffic splitter and the sensors. Our experiments suggest that early filtering reduces the number of packets to be processed by 32 percent, giving an 8 percent increase in sensor performance, locality buffers improve sensor performance by 10-18 percent, while cumulative acknowledgments improve performance by 50-90 percent. We have also developed a prototype active splitter on an IXP1200 network processor and show that the cost of the proposed approach is reasonable.

Real-time intrusion detection for high-speed networks

Network-based intrusion detection systems (NIDSs) frequently have problems with handling heavy traffic loads in real-time, which result in packet loss and false negatives. This paper presents a high-performance network intrusion detection system, called HPMonitor, which combines a high-efficiency detection engine and a load-balancing device to address these problems. The paper describes HPMonitor's system architecture, discusses a flow-based dynamic load-balancing algorithm called dynamic least load first (DLLF) algorithm, and introduces a new multi-pattern string matching algorithm called shift max algorithm (SMA). The test results reveal that the DLLF algorithm is an effective balancing algorithm for NIDS. Meanwhile, the experimental results show that the SMA algorithm is faster in searching large sets of patterns when compared with other algorithms, and its performance is affected little when the patterns set number increases.

Architecture for a hardware-based, TCP/IP content-processing system

The transmission control protocol is the workhorse protocol of the Internet. Most of the data passing through the Internet transits the network using TCP layered atop the Internet protocol (IP). Monitoring, capturing, filtering, and blocking traffic on high-speed Internet links requires the ability to directly process TCP packets in hardware. High-speed network intrusion detection and prevention systems guard against several types of threats. As the gap between network bandwidth and computing power widens, improved microelectronic architectures are needed to monitor and filter network traffic without limiting throughput. To address these issues, we've designed a hardware-based TCP/IP content-processing system that supports content scanning and flow blocking for millions of flows at gigabit line rates. The TCP splitter2 technology was previously developed to monitor TCP data streams, sending a consistent byte stream of data to a client application for every TCP data flow passing through the circuit. The content-scanning engine can scan the payload of packets for a set of regular expressions. The new TCP-based content-scanning engine integrates and extends the capabilities of the TCP splitter and the old content-scanning engine. IP packets travel to the TCP processing engine from the lower-layer-protocol wrappers. Hash tables are used to index memory that stores each flow's state.