KDD Intrusion Detection Dataset Research Articles

Classifier Selection in Resource Limited Hardware: Decision Analysis and Resolution Approach

Digitalization, Industry 4.0 and Internet of things (IoT) have become more popular in the recent years. Most of these systems depend on micro-controllers and sensors. These micro-controllers and sensors are mostly cheap, low RAM and low CPU systems; thus, they are resource constrained environments. In this study, a supervised learning classifier comparison technique suitable for resource constrained environments is proposed. This technique, Decision Analysis and Resolution (DAR), is originated in the domain of Software Engineering. First, DAR is explained using an example of car buying scenario. Then 11 off-the-shelf classifiers are compared using DAR for low RAM and less powerful CPU environments in an intrusion detection scenario. This scenario simulated on well-known KDD99 intrusion detection dataset. All the experiments are realized using python scikit-learn package. According to the experiments, Decision Tree classifier is the most suitable to implement for resource constrained environments with a small lead. Results for the other three classifiers (Bagging, Multi Layer Perceptron, Random Forest) are also very similar. To aid the reproducibility of the experiments, the whole source code of the study is provided in the popular open source repository https://github.com/ati-ozgur/classifier-comparison-using-DAR.

Implementation of Feature Selection using Information Gain Algorithm and Discretization with NSL-KDD Intrusion Detection System

Feature selection is one of the research on data mining for datasets that have relatively many attributes. Eliminating some attributes that are irrelevant to the label class will be able to improve the performance of the classification algorithm. The Information Gain algorithm is one of the algorithms for searching for features that are irrelevant to the label class. This algorithm uses wrapper techniques to eliminate irrelevant attributes. This research aims to implement feature selection using the Information Gain algorithm against the NSL KDD intrusion detection dataset which has a large number of relative attributes. The dataset of the selected attribute will be performed by a classification algorithm so that an attribute reduction can improve the compute process and improve the accuracy of the algorithm model used.

Logistic regression in large rare events and imbalanced data: A performance comparison of prior correction and weighting methods

AbstractThe purpose of this study is to use the truncated Newton method in prior correction logistic regression (LR). A regularization term is added to prior correction LR to improve its performance, which results in the truncated‐regularized prior correction algorithm. The performance of this algorithm is compared with that of weighted LR and the regular LR methods for large imbalanced binary class data sets. The results, based on the KDD99 intrusion detection data set, and 6 other data sets at both the prior correction and the weighted LRs have the same computational efficiency when the truncated Newton method is used in both of them. A higher discriminative performance, however, resulted from weighting, which exceeded both the prior correction and the regular LR on nearly all the data sets. From this study, we conclude that weighting outperforms both the regular and prior correction LR models in most data sets and it is the method of choice when LR is used to evaluate imbalanced and rare event data.

English

An intrusion detection system (IDS) is an application platform that records system fuss for malicious activities and management of these activities. It is a malicious attack identification technique. It allows identifying Denial of service (DoS), User to root (U2R), Remote to login (R2L) and Probe attack. There are several communication mechanism are rely on the networking today so the need of data security is important concern. The main aim of this paper is to focusing on improving attack detection rates. For this we have analyzed the KDD intrusion detection dataset. The main method for approaching this detection and analysis are data mining and optimization techniques for this study.

A Novel High Dimensional and High Speed Data Streams Algorithm: HSDStream

This paper presents a novel high speed clustering scheme for high-dimensional data stream. Data stream clustering has gained importance in different applications, for example, network monitoring, intrusion detection, and real-time sensing. High dimensional stream data is inherently more complex when used for clustering because the evolving nature of the stream data and high dimensionality make it non-trivial. In order to tackle this problem, projected subspace within the high dimensions and limited window sized data per unit of time are used for clustering purpose. We propose a High Speed and Dimensions data stream clustering scheme (HSDStream) which employs exponential mov-ing averages to reduce the size of the memory and speed up the processing of projected subspace data stream. It works in three steps: i) initialization, ii) real-time maintenance of core and outlier micro-clusters, and iii) on-demand offline generation of the final clusters. The proposed algorithm is tested against high dimensional density-based projected clustering (HDDStream) for cluster purity, memory usage, and the cluster sensitivity. Experi-mental results are obtained for corrected KDD intrusion detection dataset. These results show that HSDStream outperforms the HDDStream in all performance metrics, especially, the memory usage and the processing speed.

Multi-Tier Framework using Sugeno Fuzzy Inference System with Swarm Intelligence Techniques for Intrusion Detection

An intrusion detection has a key role in network security that classifies the system activities as normal or suspicious (Anomaly). An intrusion detection system must consistently detect malicious activities in a network and must perform efficiently to manage with the large amount of network traffic. The main objective of this paper is to analysis two issues such as accuracy and efficiency of the system by a novel method of incorporating swarm intelligence with data mining algorithm for feature reduction. The accuracy of the system then can be achieved by several soft computing techniques as Sugeno fuzzy inference system and simplified swarm optimization. The high efficiency can be achieved by Multi-tier approach. The proposed system uses Multi-tier-sugeno fuzzy inference system for fuzzy rule generation that effectively identifying the intrusion activities within a network, Finally, to obtain best result Simplified swarm optimization algorithm are used to optimize the structure of the fuzzy decision engine. The experimentation and evaluation of the proposed method were performed on NSL KDD intrusion detection dataset that shows best accuracy and efficiency than other methods and can easily detect whether the network data are normal or under attack. Keywords: Intrusion detection, Multi-Tier approach, NSL-KDD, Random Forest, Simplified Swarm Optimization, Sugeno Fuzzy inference system

Entropy Based Fuzzy Rule Weighting for Hierarchical Intrusion Detection

Predicting dierent behaviors in computer networks is the sub- ject of many data mining researches. Providing a balanced Intrusion Detec- tion System (IDS) that directly addresses the trade-o between the ability to detect new attack types and providing low false detection rate is a funda- mental challenge. Many of the proposed methods perform well in one of the two aspects, and concentrate on a subset of system requirements. There are many non-functional requirements for an applicable and practical IDS. The process should be online, incremental and adaptive to ever changing behaviors of normal users and attackers. Moreover providing comprehensive and inter- active IDS could both, enhance the performance of the system and extend the knowledge of domain experts. In this paper, we propose a fuzzy rule-based classication system using a hierarchical rule learning method. In each stage of the hierarchy, a set of rules with certain length of antecedent are investigated. A novel rule weighting method, based on the entropy measure, determines the appropriateness of each rule. The experimental results on KDD99 intrusion detection dataset show the eectiveness of the proposed method in tackling the tradeo between accuracy and comprehensibility of fuzzy rule-based systems. Although the dimension of antecedents is not limited, the resultant rule-base contains a small number of complex rules, which are essential to reach the desired accuracy.

A FUZZY BASED DIVIDE AND CONQUER ALGORITHM FOR FEATURE SELECTION IN KDD INTRUSION DETECTION DATASET

This paper provides a fuzzy logic based divide and conquer algorithm for feature selection and reduction among large feature set of KDD intrusion detection data set, since a reduced feature set will help to evolve better mining rules.This algorithm introduces a fuzzy idea of dividing the normal record by attacks records or vice-versa, and then considers the feature sets for every attack type separately. Actually, this algorithm is applied on KDD CUP 99 dataset having 37 attack types and selecting important feature among 41 feature of KDD dataset. The selected features are used in TANAGRA [11, 12] data mining tool to classify the dataset (i.e. KDD 99) for every attack vs. normal using various classification algorithms [5, 6]. The result for feature selection and classification shows a reduced set and maximized classification rate respectively.

ASSOCIATION RULE MINING FOR KDD INTRUSION DETECTION DATA SET

Network intrusion detection includes a set of malicious actions that compromise the integrity, confidentiality and availability of information resources. Several techniques for mining rules from KDD intrusion detection dataset [10] enables to identify attacks in the network. But little research has been done to determine the association patterns that exist between the attributes in the dataset. This paper focuses on the association rule mining in KDD intrusion dataset. Since the dataset constitutes different kinds of data like binary, discrete & continuous data, same technique cannot be applied to determine the association patterns. Hence, this paper uses varying techniques for each type of data. The proposed method is used to generate attack rules that will detect the attacks in network audit data using anomaly detection. Rules are formed depending upon various attack types. For binary data, Apriori approach is used to eliminate the non-frequent item set from the rules and for discrete and continuous value the proposed techniques are used. The paper concludes with experimental results.

Multi-class pattern classification using single, multi-dimensional feature-space feature extraction evolved by multi-objective genetic programming and its application to network intrusion detection

In this paper we investigate using multi-objective genetic programming to evolve a feature extraction stage for multiple-class classifiers. We find mappings which transform the input space into a new, multi-dimensional decision space to increase the discrimination between all classes; the number of dimensions of this decision space is optimized as part of the evolutionary process. A simple and fast multi-class classifier is then implemented in this multi-dimensional decision space. Mapping to a single decision space has significant computational advantages compared to k-class-to-2-class decompositions; a key design requirement in this work has been the ability to incorporate changing priors and/or costs associated with mislabeling without retraining. We have employed multi-objective optimization in a Pareto framework incorporating solution complexity as an independent objective to be minimized in addition to the main objective of the misclassification error. We thus give preference to simpler solutions which tend to generalize well on unseen data, in accordance with Occam's Razor. We obtain classification results on a series of benchmark problems which are essentially identical to previous, more complex decomposition approaches. Our solutions are much simpler and computationally attractive as well as able to readily incorporate changing priors/costs. In addition, we have also applied our approach to the KDD-99 intrusion detection dataset and obtained results which are highly competitive with the KDD-99 Cup winner but with a significantly simpler classification framework.

WEIGHTED THRESHOLD-BASED CLUSTERING FOR INTRUSION DETECTION SYSTEMS

Signature-based intrusion detection systems look for known, suspicious patterns in the input data. In this paper we explore compression of labeled empirical data using threshold-based clustering with regularization. The main target of clustering is to compress training dataset to the limited number of signatures, and to minimize the number of comparisons that are necessary to determine the status of the input event as a result. Essentially, the process of clustering includes merging of the clusters which are close enough. As a consequence, we will reduce original dataset to the limited number of labeled centroids. In a complex with k-nearest-neighbor (kNN) method, this set of centroids may be used as a multi-class classifier. Clearly, different attributes have different importance depending on the particular training database and given cost matrix. This importance may be regulated in the definition of the distance using linear weight coefficients. The paper introduces special procedure to estimate above weight coefficients. The experiments on the KDD-99 intrusion detection dataset have confirmed the effectiveness of the proposed methods.

Threshold-based clustering with merging and regularization in application to network intrusion detection

Signature-based intrusion detection systems look for known, suspicious patterns in the input data. In this paper we explore compression of labeled empirical data using threshold-based clustering with regularization. The main target of clustering is to compress training dataset to the limited number of signatures, and to minimize the number of comparisons that are necessary to determine the status of the input event as a result. Essentially, the process of clustering includes merging of the clusters which are close enough. As a consequence, we will reduce original dataset to the limited number of labeled centroids. In a complex with k-nearest-neighbor ( kNN) method, this set of centroids may be used as a multi-class classifier. The experiments on the KDD-99 intrusion detection dataset have confirmed effectiveness of the above procedure.

Training Genetic Programming on Half a Million Patterns: An Example From Anomaly Detection

The hierarchical RSS-DSS algorithm is introduced for dynamically filtering large datasets based on the concepts of training pattern age and difficulty, while utilizing a data structure to facilitate the efficient use of memory hierarchies. Such a scheme provides the basis for training genetic programming (GP) on a data set of half a million patterns in 15 min. The method is generic, thus, not specific to a particular GP structure, computing platform, or application context. The method is demonstrated on the real-world KDD-99 intrusion detection data set, resulting in solutions competitive with those identified in the original KDD-99 competition, while only using a fraction of the original features. Parameters of the RSS-DSS algorithm are demonstrated to be effective over a wide range of values. An analysis of different cost functions indicates that hierarchical fitness functions provide the most effective solutions.

Why machine learning algorithms fail in misuse detection on KDD intrusion detection data set

A large set of machine learning and pattern classification algorithms trained and tested on KDD intrusion detection data set failed to identify most of the user-to-root and remote-to-local attacks, as reported by many researchers in the literature. In light of this observation, this paper aims to expose the deficiencies and limitations of the KDD data set to argue that this data set should not be used to train pattern recognition or machine learning algorithms for misuse detection for these two attack categories. Multiple analysis techniques are employed to demonstrate, both objectively and subjectively, that the KDD training and testing data subsets represent dissimilar target hypotheses for user-to-root and remote-to-local attack categories. These techniques consisted of switching the roles of original training and testing data subsets to develop a decision tree classifier, cross-validation on merged training and testing data subsets, and qualitative and comparative analysis of rules generated independently on training and testing data subsets through the C4.5 decision tree algorithm. Analysis results clearly suggest that no pattern classification or machine learning algorithm can be trained successfully with the KDD data set to perform misuse detection for user-to-root or remote-to-local attack categories. It is further noted that the analysis techniques employed to assess the similarity between the two target hypotheses represented by the training and the testing data subsets can readily be generalized to data set pairs in other problem domains.