IDS Dataset Research Articles

Hybrid dung beetle optimization based dimensionality reduction with deep learning based cybersecurity solution on IoT environment

The Internet of Things (IoT) interconnects various devices and objects through the Internet to interact with corresponding devices or machines. Now, consumers can purchase many internet-connected products, from automobiles to refrigerators. Extending network capacities to every aspect of life can save money and time, increase efficiency, and enable greater access to digital experiences. Cybersecurity analysts often refer to this as increasing the attack surface from which hackers can benefit. Implementing the proper security measures is crucial since IoT devices can be vulnerable to cyberattacks and are often built with limited security features. Securing IoT devices involves implementing security measures and best practices to secure them from potential vulnerabilities and threats. Deep learning (DL) models have recently analyzed the network pattern for detecting and responding to possible intrusions, improving cybersecurity with advanced threat detection abilities. Therefore, this study presents a new Hybrid Dung Beetle Optimization-based Dimensionality Reduction with a Deep Learning-based Cybersecurity Solution (HDBODR-DLCS) method on the IoT network. The primary goal of the HDBODR-DLCS technique is to perform dimensionality reduction with a hyperparameter tuning process for enhanced detection results. In the primary stage, the HDBODR-DLCS technique involves Z-score normalization to measure the input dataset. The HDBO model is used for dimensionality reduction, which mainly selects the relevant features and discards the irrelevant features. Besides, intrusions are detected using the attention bidirectional recurrent neural network (ABiRNN) model. Finally, an artificial rabbits optimization (ARO) based hyperparameter tuning process is performed, enhancing the overall classification performance. The empirical analysis of the HDBODR-DLCS method is tested under the benchmark IDS dataset. The simulation outcomes indicated the HDBODR-DLCS method's improved abilities over existing approaches.

Performance Study on the Use of Genetic Algorithm for Reducing Feature Dimensionality in an Embedded Intrusion Detection System

Intrusion Detection Systems play a crucial role in a network. They can detect different network attacks and raise warnings on them. Machine Learning-based IDSs are trained on datasets that, due to the context, are inherently large, since they can contain network traffic from different time periods and often include a large number of features. In this paper, we present two contributions: the study of the importance of Feature Selection when using an IDS dataset, while striking a balance between performance and the number of features; and the study of the feasibility of using a low-capacity device, the Nvidia Jetson Nano, to implement an IDS. The results, comparing the GA with other well-known techniques in Feature Selection and Dimensionality Reduction, show that the GA has the best F1-score of 76%, among all feature/dimension sizes. Although the processing time to find the optimal set of features surpasses other methods, we observed that the reduction in the number of features decreases the GA processing time without a significant impact on the F1-score. The Jetson Nano allows the classification of network traffic with an overhead of 10 times in comparison to a traditional server, paving the way to a near real-time GA-based embedded IDS.

Performance study of fuzzy C-mean clustering algorithm based on local density in network security

The development and utilization of network big data is also accompanied by data theft and destruction, so the monitoring of network security is particularly important. Based on this, the study applies the fuzzy C-mean clustering algorithm to the network security model, however, the algorithm has major defects in discrete data processing and the influence of feature weights. Therefore, the study introduces the concept of local density and optimizes the initial clustering center to solve its sensitive defects as well as empirical limitations; at the same time, the study introduces the adaptive methods of fuzzy indicators and feature weighting, and uses the concepts such as fuzzy center-of-mass distribution to avoid problems such as the model converging too fast and not being able to handle discrete data. Finally, the study does a simulation analysis of the performance of each module, and the comparison of the overall algorithm with the rest of the models. The experimental results show that in the comparison of the overall algorithm, its false detection rate decreases by 8.57% in the IDS Dataset dataset, compared to the particle swarm algorithm. Therefore, the adaptive weighted fuzzy C-Means algorithm based on local density proposed in the study can effectively improve the network intrusion detection performance.

Optimizing deep learning-based intrusion detection in cloud computing environment with chaotic tunicate swarm algorithm

Intrusion Detection (ID) in cloud environments is vital to maintain the safety and integrity of data and resources. However, the presence of class imbalance, where normal samples significantly outweigh intrusive instances, poses a challenge in constructing a potential ID system. Deep Learning (DL) methods, with their capability to automatically study complex patterns and features, present a promising solution in various ID tasks. Such methods can automatically learn intricate features and patterns from the input dataset, making them suitable for detecting anomalies and finding intrusions in cloud environments. Therefore, this study proposes a Class Imbalance Data Handling with an Optimal Deep Learning-Based Intrusion Detection System (CIDH-ODLIDS) in a cloud computing atmosphere. The CIDH-ODLIDS technique leverages optimal DL-based classification and addresses class imbalance. Primarily, the CIDH-ODLIDS technique preprocesses the input data using a Z-score normalization approach to ensure data quality and consistency. To handle class imbalance, the CIDH-ODLIDS technique employs oversampling techniques, particularly focused on synthetic minority oversampling techniques such as Adaptive Synthetic (ADASYN) sampling. ADASYN generates synthetic instances for the minority class depending on the available data instances, effectively balancing the class distribution and mitigating the impact of class imbalance. For the ID process, the CIDH-ODLIDS technique utilizes a Fuzzy Deep Neural Network (FDNN) model, and its tuning procedure is performed using the Chaotic Tunicate Swarm Algorithm (CTSA). CTSA is employed to choose the learning rate of the FDNN methods optimally. The experimental assessment of the CIDH-ODLIDS method is extensively conducted on three IDS datasets. The comprehensive comparison results confirm the superiority of the CIDH-ODLIDS algorithm over existing techniques.

CNN-GRU-FF: a double-layer feature fusion-based network intrusion detection system using convolutional neural network and gated recurrent units

Identifying and preventing malicious network behavior is a challenge for establishing a secure network communication environment or system. Malicious activities in a network system can seriously threaten users’ privacy and potentially jeopardize the entire network infrastructure and functions. Furthermore, cyber-attacks have grown in complexity and number due to the ever-evolving digital landscape of computer and network devices in recent years. Analyzing network traffic using network intrusion detection systems (NIDSs) has become an integral security measure in modern networks to identify malicious and suspicious activities. However, most intrusion detection datasets contain imbalance classes, making it difficult for most existing classifiers to achieve good performance. In this paper, we propose a double-layer feature extraction and feature fusion technique (CNN-GRU-FF), which uses a modified focal loss function instead of the traditional cross-entropy to handle the class imbalance problem in the IDS datasets. We use the NSL-KDD and UNSW-NB15 datasets to evaluate the effectiveness of the proposed model. From the research findings, it is evident our CNN-GRU-FF method obtains a detection rate of 98.22% and 99.68% using the UNSW-NB15 and NSL-KDD datasets, respectively while maintaining low false alarm rates on both datasets. We compared the proposed model’s performance with seven baseline algorithms and other published methods in literature. It is evident from the performance results that our proposed method outperforms the state-of-the-art network intrusion detection methods.

Benchmarking the benchmark — Comparing synthetic and real-world Network IDS datasets

Network Intrusion Detection Systems (NIDSs) are an increasingly important tool for the prevention and mitigation of cyber attacks. Over the past years, a lot of research efforts have aimed at leveraging the increasingly powerful models of Machine Learning (ML) for this purpose. A number of labelled synthetic datasets have been generated and made publicly available by researchers, and they have become the benchmarks via which new ML-based NIDS classifiers are being evaluated. Recently published results show excellent classification performance with these datasets, increasingly approaching 100 percent performance across key evaluation metrics such as Accuracy, F1 score, AUC, etc. Unfortunately, we have not yet seen these excellent academic research results translated into practical NIDS systems with such near-perfect performance. This motivated our research presented in this paper, where we analyse the statistical properties of the benign traffic in three of the more recent and relevant NIDS datasets, (CIC_IDS, UNSW_NB15, TON_IOT), by converting them into a common flow format. As a comparison, we consider two datasets obtained from real-world production networks, one from a university network and one from a medium size Internet Service Provider (ISP). Our results show that the two real-world datasets are quite similar among themselves in regards to most of the considered statistical features. Equally, the three synthetic datasets are also relatively similar within their group. However, and most importantly, our results show a distinct difference of most of the considered statistical features between the three synthetic datasets and the two real-world datasets. Since ML relies on the basic assumption of training and test datasets being sampled from the same distribution, this raises the question of how well the performance results of ML-classifiers trained on the considered synthetic datasets can translate and generalise to real-world networks. We believe this is an interesting and relevant question which provides motivation for further research in this space.

From Bytes to Insights: A Systematic Literature Review on Unraveling IDS Datasets for Enhanced Cybersecurity Understanding

From Bytes to Insights: A Systematic Literature Review on Unraveling IDS Datasets for Enhanced Cybersecurity Understanding

HIDM: Hybrid Intrusion Detection Model for Industry 4.0 Networks Using an Optimized CNN-LSTM with Transfer Learning.

Industrial automation systems are undergoing a revolutionary change with the use of Internet-connected operating equipment and the adoption of cutting-edge advanced technology such as AI, IoT, cloud computing, and deep learning within business organizations. These innovative and additional solutions are facilitating Industry 4.0. However, the emergence of these technological advances and the quality solutions that they enable will also introduce unique security challenges whose consequence needs to be identified. This research presents a hybrid intrusion detection model (HIDM) that uses OCNN-LSTM and transfer learning (TL) for Industry 4.0. The proposed model utilizes an optimized CNN by using enhanced parameters of the CNN via the grey wolf optimizer (GWO) method, which fine-tunes the CNN parameters and helps to improve the model's prediction accuracy. The transfer learning model helps to train the model, and it transfers the knowledge to the OCNN-LSTM model. The TL method enhances the training process, acquiring the necessary knowledge from the OCNN-LSTM model and utilizing it in each next cycle, which helps to improve detection accuracy. To measure the performance of the proposed model, we conducted a multi-class classification analysis on various online industrial IDS datasets, i.e., ToN-IoT and UNW-NB15. We have conducted two experiments for these two datasets, and various performance-measuring parameters, i.e., precision, F-measure, recall, accuracy, and detection rate, were calculated for the OCNN-LSTM model with and without TL and also for the CNN and LSTM models. For the ToN-IoT dataset, the OCNN-LSTM with TL model achieved a precision of 92.7%; for the UNW-NB15 dataset, the precision was 94.25%, which is higher than OCNN-LSTM without TL.

A novel‐cascaded ANFIS‐based deep reinforcement learning for the detection of attack in cloud IoT‐based smart city applications

SummaryVast usages of Internet of Things (IoT) devices in various smart applications have laid a foundation for the evolution of modern smart cities. The increasing dependency of smart city applications on communication and information technologies enhances operational efficiency, sustainability, and automation of city services. However, due to the heterogeneous nature of IoT devices, the network faces critical security issues while executing continued network operations and services, particularly by cyber‐attacks. One of the predominant and rampant cyber‐attacks in smart city applications is botnet attacks. Therefore, a novel deep learning model for the detection and isolation of cyber‐attacks is proposed in the cloud IoT‐based smart city applications to protect against such cyber‐attacks. The proposed framework utilizes two different modules to automatically detect and isolate the malicious traffic emanating from compromised IoT devices with more efficiency. Here, two different datasets namely the IoT network intrusion and the ISCX 2012 IDs datasets are utilized for the evaluation of the proposed framework. In the first phase, the compromised device which communicates malicious network traffics through the network is identified using a cascaded adaptive neuro‐fuzzy inference system (CANFIS). After detection, IP address of abnormal traffic is recorded and informed to the system administrator. In the second phase, communication pathways of compromised devices with other normal devices are blocked and the compromised devices are isolated from the network using the modified deep reinforcement learning (MDRL) approach. The analytic result shows that the proposed framework achieves a greater accuracy rate of about 98.7% as compared to other state‐of‐art methods.

Data Generative Model to Detect the Anomalies for IDS Imbalance CICIDS2017 Dataset

The system of intrusion detection dataset enables machine learning to recognize attack activity in the network. The intrusion, however, is naturally imbalanced, most of the traffic is normal traffic. Moreover, a certain attack is more popular than others. Therefore, the number of cases is highly imbalanced with the majority of attacks dominated by Distributed Denial of Services (DDoS), Denial of Service Hulk (DoS_Hulk), and PortScan more than 4.5% of attacks data. While the minority attack such as DoS_goldeneye, DoS_slowloris, DoS_slowhttptest, Web Attacks, Infiltration, Bot, and Heartbleed was only recorded in less than 1% of attack data. We propose data generative model (DGM) using the Conditional Generative Adversarial Network (CGAN) to improve the class of minorities of the IDS dataset. In this study, we tested the performance of the Data Generative Model based on CGAN in the CICIDS2017 dataset. There are new attacks in this dataset, including Bot, Web_attacks, Infiltration and Heartbleed. According to our experiments, the model successfully detect new attacks and improves the weighted f1-score by 99,92% compared to that of achievers by existing methods using the CICIDS2017 dataset.

DETECTION OF BOTNET ATTACK IN INTERNET OF THINGS USING FEATURE SELECTION ENABLE MACHINELERANING TECHNIQUE

The Internet of Things (IoT) is very rapidly developing with millions of devices that are usedin the smart home ,smart city ,and many other smart systems for education and so on . On the other side , attackers are mostly targeting these devices. After infecting malware attacks on these devices ,they become bots that are controlled by attackers ,and these will be targeted to these organization not only for stealing important information but also for breaking down of the network. Although certain security measures have been created to guard against cyber-attacks, themajority of these systems are rule-based. Additionally, malware attackers may be able to avoid the formal rulebased detection. Theabsence of earlier detection strategies is thereforefilled by the machine learningbased detection scheme. The public IDS dataset N-baloT andmachine learning techniques like the CART algorithm provide the foundation of thesuggested detection architecture. Machine learning models based on four classifiers are built :Naïve bayes ,K-nearest neighbour ,Support vector Machine ,and decision trees. Using 82,000 records from UNSW-NB15 dataset, the decision tree model has yieldedthe best overall result with 99,89% testing accuracy, 100% precision, 100% recall, and 100% F-score in detecting botnet attacks.

Automatic False Alarm Detection Based on XAI and Reliability Analysis

Many studies attempt to apply artificial intelligence (AI) to cyber security to effectively cope with the increasing number of cyber threats. However, there is a black box problem such that it is difficult to understand the basis for AI prediction. False alarms for malware or cyberattacks can cause serious side effects. Due to this limitation, all AI predictions must be confirmed by an expert, which is a considerable obstacle to AI expansion. Compared to the increasing number of cyberattack alerts, the number of alerts that can be analyzed by experts is limited. This paper provides explainability through an interpretation of AI prediction results and a reliability analysis of AI predictions based on explainable artificial intelligence (XAI). In addition, we propose a method for screening high-quality data that can efficiently detect false predictions based on reliability indicators. Through this, even a small security team can quickly respond to false predictions. To validate the proposed method, experiments were conducted using the IDS dataset and the malware dataset. AI errors were detected better than they could be by the existing AI models, with about 262% in the IDS dataset and 127% in the malware dataset from the top 10% of analysis targets. Therefore, the ability to respond to cyberattacks can be improved using the proposed method.

A Deep Learning Ensemble Approach to Detecting Unknown Network Attacks

The majority of the intrusion detection solutions proposed using machine learning and deep learning approaches are based on known attack classes only. Comprehensive threat detection systems should consider both known and unknown attacks. Rapidly changing network environment and the advanced tools and techniques used by adversaries to launch new sophisticated attacks highlight a growing need to build intrusion detection systems that are more realistic, diverse, and robust to detect known and unknown attacks. We employed deep-learning models in our experiments to detect unknown threats, never introduced before to the model. This paper also studied the bias issues in connection with unknown threats detection. Many recent research studies based on conventional machine learning may report biased results and restricted training due to relying only on a single dataset; thus, there are existing threats that the model is unaware of, although the model may have high accuracy (in the known territories). This study presents a realistic IDS approach in which a deep learning classifiers' ensemble is trained on four benchmark IDS datasets for testing the unknown attack instances. Specifically, the model has no prior knowledge of some labels and traffic patterns in those experiments. The architecture proposed builds a deep learning ensemble using classifiers well-known to process and produce good results for sequential data. Our empirical results indicate that the proposed ensemble model can detect a range of unknown attacks with reasonable performance measures and a practical approach towards building a comprehensive IDS solution.

COMPARATIVE ANALYSIS OF THE K-NEAREST NEIGHBOR ALGORITHM ON VARIOUS INTRUSION DETECTION DATASETS

Security in computer networks can be vulnerable, this is because we have weaknesses in making security policies, weak computer system configurations, or software bugs. Intrusion detection is a mechanism for securing computer networks by detecting, preventing, and blocking illegal attempts to access confidential information. The IDS mechanism is designed to protect the system and reduce the impact of damage from any attack on a computer network for violating computer security policies including availability, confidentiality, and integrity. Data mining techniques have been used to obtain useful knowledge from the use of IDS datasets. Some IDS datasets that are commonly used are Full KDD, Corrected KDD99, NSL-KDD, 10% KDD, UNSW-NB15, Caida, ADFA Windows, and UNM have been used to get the accuracy rate using the k-Nearest Neighbors algorithm (k-NN). The latest IDS dataset provided by the Canadian Institute of Cybersecurity contains most of the latest attack scenarios named the CICIDS2017 dataset. A preliminary experiment shows that the approach using the k-NN method on the CICIDS2017 dataset successfully produces the highest average value of intrusion detection accuracy than other IDS datasets.

An Innovative Perceptual Pigeon Galvanized Optimization (PPGO) Based Likelihood Naïve Bayes (LNB) Classification Approach for Network Intrusion Detection System

ABSTRACTIntrusion detection and classification have gained significant attention recently due to the increased utilization of networks. For this purpose, there are different types of Network Intrusion Detection System (NIDS) approaches developed in the conventional works, which mainly focus on identifying the intrusions from the datasets with the help of classification techniques. Still, it is limited by the significant problems of inefficiency in handling large dimensional datasets, high computational complexity, false detection, and more time consumption for training the models. To solve these problems, this research intends to develop an innovative clustering-based classification methodology to precisely detect intrusions from the different types of IDS datasets. Here, the most recent and extensively used IDS datasets such as NSL-KDD, CICIDS, and Bot-IoT have been employed for detecting intrusions. Data preprocessing has been performed to normalize the dataset to eliminate irrelevant attributes and organize the features. Then, the data separation is applied by forming the clusters by using an intelligent Anticipated Distance-based Clustering (ADC) incorporated with the Density-Based Spatial clustering of applications with noise (DBScan) algorithm. It helps to find the distance and density measures for grouping the attributes into the clusters, which increases the efficiency of classification. Here, the most suitable optimal parameters are selected using the Perpetual Pigeon Galvanized Optimization (PPGO) technique. The extracted features are used for training and testing the dataset samples.Consequently, the Likelihood Naove Bayes (LNB) classification approach is implemented to accurately predict the classified label as to whether normal or attack. During the evaluation, the performance of the proposed IDS framework is validated and compared using various evaluation metrics. Theresults show that the proposed ADC-DBScan-LNB model outperforms the other techniques with improved performance outcomes.

Detection of Network Attacks using Machine Learning: A New Approach

Abstract: The Cyber-attacks become the most important security problems in the today’s world. With the increase in use of computing resources connected to the Internet like computers, mobiles, sensors, IoTs in networks, Big Data, Web Applications/Server, Clouds and other computing resources, hackers and malicious users are planning new ways of network intrusions. Many techniques have been developed to detect these intrusions which are based on data mining and machine learning methods. These intrusions detection techniques have been applied on various IDS datasets. UNSW-NB15 is the latest dataset. This data set contains different modern attack types and wide varieties of real normal activities. In this paper, we compare Naïve Bays algorithm with proposed probability based supervised machine learning algorithms using reduced UNSW NB15 dataset. Keywords: UNSW NB-15, Machine Learning, Naïve Bayes, All to Single (AS) features probability Algorithm

Deep Learning-Based Framework for the Detection of Cyberattack Using Feature Engineering

Digital systems are changing to security systems in contemporary days. It is time for the digital system to have sufficient security to defend against threats and attacks. The intrusion detection system can identify an anomaly from an external or internal source in the network system. Many kinds of threats are present, that is, active and passive. These dangers could lead to anomalies in the system by which data can be attacked and taken by attackers from the beginning to the destination. Machine learning nowadays is a developing topic; its applications are wide. We can forecast the future through machine learning and classify the right class. In this paper, we employed the new binary and multiclass classification model of Convolutional Neural Networks (CNNs) to identify the anomaly of the network system. In this respect, we used the NSLKDD dataset. Our model uses a Convolutional Neural Network (CNN) to conduct binary and multiclass classification. In both datasets, we build a DL-based DoS detection model. We focus on the DoS category in the most extensively used IDS dataset, KDD. As the name implies, CNN is the most extensively used the DL model for image recognition. Adding a pooling layer to the convolution layer minimizes the size of the feature data extracted from the image while maintaining I/O and spatial information. The CNN model has shown the promising results of multiclass and binary classification in terms of validation loss of 0.0012 at 11th epochs and validation accuracy of 98% and 99%, respectively.

LONGPOP and IDS. Personal reflections on our collaboration With Kees Mandemakers.

The Leuven research team working on historical demography is grateful for their opportunity to have collaboratively and intensively worked with Professor Dr. Kees Mandemakers over an extended period of time. We wish him a wonderful emeritus status, not only in academia, but also in the warm nest of his family, relatives, children, and grandchildren. The three of us have known Kees for some time, but most closely since 2014, when we became formally engaged as project partners under the so-called LONGPOP-project, an EU funded Marie Curie grant, named Methodologies and Data mining techniques for the analysis of Big Data based on Longitudinal Population and Epidemiological Registers. The importance of our close professional relationship is best demonstrated by our work in producing the COR*-IDS 2020 database. The historical demographic dataset for the Antwerp arrondissement, a letter sample COR*-2010, recorded total sample size of +/- 33,000 residents of Antwerp for nearly seven decades and was already available. The LONGPOP project began in Autumn 2016, setting in motion a collaboration between ourselves at KU Leuven and Kees and his colleagues at the IISH. From the outset, we had purposefully worked closely with Kees' team, utilising their premier expertise in database management and the IDS towards the new release of our database, the Antwerp COR*-IDS dataset. Here we set out our recollections of that intellectual process, encompassing the personal and professional reflections of our close working relationship.

A Consolidated Decision Tree-Based Intrusion Detection System for Binary and Multiclass Imbalanced Datasets

The widespread acceptance and increase of the Internet and mobile technologies have revolutionized our existence. On the other hand, the world is witnessing and suffering due to technologically aided crime methods. These threats, including but not limited to hacking and intrusions and are the main concern for security experts. Nevertheless, the challenges facing effective intrusion detection methods continue closely associated with the researcher’s interests. This paper’s main contribution is to present a host-based intrusion detection system using a C4.5-based detector on top of the popular Consolidated Tree Construction (CTC) algorithm, which works efficiently in the presence of class-imbalanced data. An improved version of the random sampling mechanism called Supervised Relative Random Sampling (SRRS) has been proposed to generate a balanced sample from a high-class imbalanced dataset at the detector’s pre-processing stage. Moreover, an improved multi-class feature selection mechanism has been designed and developed as a filter component to generate the IDS datasets’ ideal outstanding features for efficient intrusion detection. The proposed IDS has been validated with state-of-the-art intrusion detection systems. The results show an accuracy of 99.96% and 99.95%, considering the NSL-KDD dataset and the CICIDS2017 dataset using 34 features.

Cost-Effective Valuable Data Detection Based on the Reliability of Artificial Intelligence

Many previous studies have investigated applying artificial intelligence (AI) to cyber security. Despite considerable performance advantages, AI for cyber security requires final confirmation by an analyst, e.g. malware misdetection can cause significant adverse side effects. Thus, a human analyst must check all AI predictions, which poses a major obstacle to AI expansion. This paper proposes a reliability indicator for AI prediction using explainable artificial intelligence and statistical analysis techniques. This will enable analysts with limited daily workload to focus upon valuable data, and quickly verify AI predictions. Analysts generally make decisions based on several features that they know exactly what they mean, rather than all available features. Since the proposed reliability indicator is calculated using features meaningful to analysts, it can be easily understood and hence speed final decisions. To verify the performance of the proposed method, an experiment was conducted using the IDS dataset and the malware dataset. The AI error was detected better than the existing AI model at about 114% in IDS and 95% in malware. Thus, cyberattack response could be greatly improved by adopting the proposed method.