Large-scale network intrusion detection based on distributed learning algorithm

Abstract

As network traffic bandwidth is increasing at an exponential rate, it’s impossible to keep up with the speed of networks by just increasing the speed of processors. Besides, increasingly complex intrusion detection methods only add further to the pressure on network intrusion detection (NIDS) platforms, so the continuous increasing speed and throughput of network poses new challenges to NIDS. To make NIDS usable in Gigabit Ethernet, the ideal policy is using a load balancer to split the traffic data and forward those to different detection sensors, which can analyze the splitting data in parallel. In order to make each slice contains all the evidence necessary to detect a specific attack, the load balancer design must be complicated and it becomes a new bottleneck of NIDS. To simplify the load balancer this paper put forward a distributed neural network learning algorithm (DNNL). Using DNNL a large data set can be split randomly and each slice of data is presented to an independent neural network; these networks can be trained in distribution and each one in parallel. Completeness analysis shows that DNNL’s learning algorithm is equivalent to training by one neural network which uses the technique of regularization. The experiments to check the completeness and efficiency of DNNL are performed on the KDD’99 Data Set which is a standard intrusion detection benchmark. Compared with other approaches on the same benchmark, DNNL achieves a high detection rate and low false alarm rate.