DARPA Dataset Research Articles

T-Trace: Constructing the APTs Provenance Graphs Through Multiple Syslogs Correlation

Advanced Persistent Threats (APTs) employ sophisticated and covert tactics to infiltrate target systems, leading to increased vulnerability and an elevated risk of exposure. Consequently, it is essential for us to proactively create an extensive and clearly outlined attack chain for APTs in order to effectively combat these threats. Unlike traditional malware or application threats, APTs can sidestep cyber security efforts and cause severe damage to organizations or even state security. Nonetheless, earlier methods struggle to accurately track APTs and may face a dependency explosion issue, as identifying the intricate and complex unknown malicious activities within APTs proves to be challenging. In this paper, we propose and build an approach, T-trace, which constructs the events provenance graphs by analyzing the correlations among logs. The approach precisely finds the log communities with tensor decomposition and calculates significance scores to extract the events. The APTs can be inferred by discovering the event communities and constructing the provenance graph with log correlation. In the experiment, we used DARPA data sets and launched four current practical APTs. Compared with current approaches, the results show that T-trace can efficiently reduce time cost by 90% and achieve a 92% accuracy rate in constructing the provenance graph, which can be practically applied in APTs provenance.

DDoS attack detection in SDN: Enhancing entropy‐based detection with machine learning

SummarySoftware defined network (SDN) has emerged as a new paradigm in terms of network architecture, providing flexibility, agility, and programmability to network management. These benefits boosted the SDN adoption, bringing new challenges mainly related to security, in particular, those related to Denial of Service (DoS) and Distributed Denial of Service (DDoS) attacks. The detection, prevention, and mitigation of these attacks are important since they can affect the entire network. Many current security measures use statistical techniques, as entropy, or machine learning (ML) algorithms to detect DoS and DDoS attacks. While the definition of a threshold to determine whether a traffic is an attack is not trivial in statistical techniques, ML solutions may provide better accuracy but require considerable computational resources and time to converge to a model able to detect these attacks. Trying to circumvent these limitations, current hybrid approaches either use the results from entropy as input in ML algorithms (EntropyML) or use entropy as a filter and ML algorithms to identify attacks. This work goes one step ahead and combines these techniques in a three‐step approach (EntropyMLEntropy), called ML‐Entropy, which inherits the intelligence of ML algorithms to adjust the threshold used by entropy. The proposed solution was implemented and evaluated in two datasets, the well‐known synthetic DARPA dataset and a dataset composed by traffic collected from a real‐corporate environment. Experimental results show that, in general, ML‐Entropy presents an accuracy above 99%, similar to support vector machine (SVC) and random forest (RF) algorithms, being able to converge to a detection model up to and faster than RF and SVC, respectively.

Performance Evaluation of Network Intrusion Detection System for Detecting Zero-Day Attacks: SNORT-XSS Algorithm

The main objective of Intrusion Detection and Prevention Systems is to provide a method of detecting and preventing malicious behaviors in a network system to minimize the harm caused by attackers. In this article, a survey of the techniques applied for the identification and classification of attacks based on KDD Cup’99 and DARPA data set is discussed, and from the open issues a new and a proficient method called SNORT-XSS algorithm is anticipated and implemented that can recognize and classify real time intrusions including zero day attacks. For this research, the SNORT open source tool developed by CISCO Systems was used to describe rules from the existing data collected from DARPA and KDD Cup’99 dataset. Fuzzy Reasoning system is applied to organize the rules into fuzzy sets that reduces true negative and false positive rate. The advantage of Feed Forward Neural Network with Back Propagation of Errors from Artificial Neuron Networks is considered for training, validating and testing the proposed system. The experimental results achieved by preprocessing anomalous behaviors in a network and the detection rate of zero-day attacks or novel attacks were very promising and were beyond expectations. The precision values of the proposed model were 98.93% and 98.89% respectively, and detection rate of Probe and DoS attacks were greater than 98%. The false positive and true negative rate is almost negligible. It was noticed that the best categorization was acquired at epoch numbers from 50 to 55 with a mean squared error of 0.004.

An Effective Probabilistic Technique for DDoS Detection in OpenFlow Controller

Distributed denial of service (DDoS) attacks have always been a nightmare for network infrastructure for the last two decades. Existing network infrastructure is lacking in identifying and mitigating the attack due to its inflexible nature. Currently, software-defined networking (SDN) is more popular due to its ability to monitor and dynamically configure network devices based on the global view of the network. In SDN, the control layer is accountable for forming all decisions in the network and data plane for just forwarding the message packets. The unique property of SDN has brought a lot of excitement to network security researchers for preventing DDoS attacks. In this article, for the identification of DDoS attacks in the OpenFlow controller, a probabilistic technique with a central limit theorem has been utilized. This method primarily detects resource depletion attacks, for which the DARPA dataset is used to train the probabilistic model. In different attack scenarios, the probabilistic approach outperforms the entropy-based method in terms of false negative rate (FNR). The emulation results demonstrate the efficacy of the approach, by reducing the FNR by 98% compared to 78% in the existing entropy mechanism, at 50% attack rate.

Statistical Techniques for Detecting Cyberattacks on Computer Networks Based on an Analysis of Abnormal Traffic Behavior

Represented paper is currently topical, because of year on year increasing quantity and diversity of attacks on computer networks that causes significant losses for companies. This work provides abilities of such problems solving as: existing methods of location of anomalies and current hazards at networks, statistical methods consideration, as effective methods of anomaly detection and experimental discovery of choosed method effectiveness. The method of network traffic capture and analysis during the network segment passive monitoring is considered in this work. Also, the processing way of numerous network traffic indexes for further network information safety level evaluation is proposed. Represented methods and concepts usage allows increasing of network segment reliability at the expense of operative network anomalies capturing, that could testify about possible hazards and such information is very useful for the network administrator. To get a proof of the method effectiveness, several network attacks, whose data is storing in specialised DARPA dataset, were chosen. Relevant parameters for every attack type were calculated. In such a way, start and termination time of the attack could be obtained by this method with insignificant error for some methods.

Discriminating flash crowds from DDoS attacks using efficient thresholding algorithm

Distributed Denial-of-Service attacks have been a challenge to cyberspace, as the attackers send a large number of attack packets similar to the normal traffic, to throttle legitimate flows. These attacks intentionally disrupt the services offered by the systems resulting in heavy cost. A flash crowd or flash event is an unexpected surge in the number of visitors to a particular website resulting in a sudden increase in server load. Flash crowds, which are legitimate flows, are difficult to be discriminated from Distributed Denial-of-Service attacks that are illicit flows. Effective and accurate detection of Distributed Denial of Service attacks still remains a challenge due to the difficulty in its detection and the false alerts generated in the case of flash crowds. There is a trade off between detection rate and false positive rate. This work deals with an efficient and early detection of distributed denial of service attacks and discriminates flash crowd by considering two network traffic parameters such as packet size and destination IP address. Using these traffic features two attributes are computed and its generalized entropies are calculated. The threshold is computed using the mean value of network attributes to detect the attacks. Threshold updater can automatically adjust the threshold values according to the changes in the channel conditions. The data sets used to evaluate the performance of the proposed approach are the MIT Lincoln Laboratory DARPA data set and a data set generated in a University network. Experimental results show this research approach achieves higher detection rate and lower false positives in a much reduced processing time as compared to the existing methods.

Application of Adaptive Threshold Algorithm with Selected Modified Parameters for the Detection of Flooding Based Denial-of-Service (DoS) Attack in Mobile AdHoc Network

Mobile AdHoc Networks (MANET) were self- organizing networks formed without pre-installed infrastructure. Nodes in these networks extensively relied on intermediate nodes to communicate with the destination node. Due to the limited network and host resources, Denial-of-Service (DoS) attacks were the major problem faced by MANET. Flooding based DoS attacks aimed to overwhelm certain resources which resulted in destabilizing an entire network which further denying the access of services to legitimate users. SYN flooding attack was a flooding attack launched from the transport layer of TCP/IP protocol stack in which the attacker forwarded a prodigious number of spoofed SYN packets to the victim node in order to overflow its back log queue. Fixed Threshold Algorithm (FTA) or Adaptive Threshold Algorithm (ATA) was a commonly used technique for detecting these attacks in the network in which an alarm was raised whenever the number of SYN packets increased abnormally. But these algorithms were suffered from a higher false positive rate (FPR) and lower accuracy.There as on for these limitations was that network traffic statistics were updated regularly during normal traffic conditions as well as for attack traffic. In order to overcome these limitations,a modified version of the Adaptive Threshold Algorithm was proposed.Based on the comprehensive evaluation using DARPA data set,it was shown that the present work could successfully detect flooding attacks with higher accuracy and lower false positive rate.

ENHANCED SOFT COMPUTING APPROACHES FOR INTRUSION DETECTION SCHEMES IN SOCIAL MEDIA NETWORKS

The soft computing methods play a vital role in identifying the malicious activities in the social network. The low cost solutions and the robustness provided by the soft computing in the identifying the unwanted activities make it a predominant area of research. The paper combines the soft computing techniques and frames an enhanced soft computing approach to detect the intrusion that cause security issues in the social network. The proffered method of the paper employs the enhanced soft computing technique that combines the fuzzy logic, decision tree, K means -EM and the machine learning in preprocessing, feature reduction, clustering and classification respectively to develop a security approach that is more effective than the traditional computations in identifying the misuse in the social networks. The intrusion detection system developed using the soft computing approach is tested using the KDD-NSL and the DARPA dataset to note down the security percentage, time utilization, cost and compared with the other traditional methods.

Siamese Tracking from Single Point Initialization.

Recently, we have been concerned with locating and tracking vehicles in aerial videos. Vehicles in aerial videos usually have small sizes due to use of cameras from a remote distance. However, most of the current methods use a fixed bounding box region as the input of tracking. For the purpose of target locating and tracking in our system, detecting the contour of the target is utilized and can help with improving the accuracy of target tracking, because a shape-adaptive template segmented by object contour contains the most useful information and the least background for object tracking. In this paper, we propose a new start-up of tracking by clicking on the target, and implement the whole tracking process by modifying and combining a contour detection network and a fully convolutional Siamese tracking network. The experimental results show that our algorithm has significantly improved tracking accuracy compared to the state-of-the-art regarding vehicle images in both OTB100 and DARPA datasets. We propose utilizing our method in real time tracking and guidance systems.

Efficient DDoS flood attack detection using dynamic thresholding on flow-based network traffic

Internet applications are used in various sectors as it contributes in enhancing the system usage in many respects. However, the interconnected computer systems and networks are vulnerable to very large number of attacks; Distributed Denial of Service being a major one. This paper analyses the features of network traffic and the existing algorithms to detect Distributed Denial of Service attacks and proposes an efficient statistical approach to detect the attacks based on traffic features and dynamic threshold detection algorithm. Dynamic threshold is made use of since both network activities and user’s behaviour could vary over time. The proposed algorithm extract different traffic features, calculate four attributes based on the characteristics of Distributed Denial of Service and the attack gets detected when the calculated attributes within a time interval is greater than the threshold value. MIT Lincoln Laboratory DARPA datasets and dataset developed in an university laboratory are used to validate the algorithm and the model proposed in this paper and also to measure the performance of the proposed approach. Experimental results demonstrate the improved performance of the proposed approach with significantly higher detection rate and accuracy and lesser processing time compared to the existing methods.

Implementation of Network Intrusion Detection System Using Soft Computing Algorithms (Self Organizing Feature Map and Genetic Algorithm)

In today’s world, computer network is evolving very rapidly. Most public or/and private companies set up their own local networks system for the purpose of promoting communication and data sharing within the companies. Unfortunately, their data and local networks system are under risks. With the advanced computer networks, the unauthorized users attempt to access their local networks system so as to compromise the integrity, confidentiality and availability of resources. Multiple methods and approaches have to be applied to protect their data and local networks system against malicious attacks. The main aim of our paper is to provide an intrusion detection system based on soft computing algorithms such as Self Organizing Feature Map Artificial Neural Network and Genetic Algorithm to network intrusion detection system. KDD Cup 99 and 1998 DARPA dataset were employed for training and testing the intrusion detection rules. However, GA’s traditional Fitness Function was improved in order to evaluate the efficiency and effectiveness of the algorithm in classifying network attacks from KDD Cup 99 and 1998 DARPA dataset. SOFM ANN and GA training parameters were discussed and implemented for performance evaluation. The experimental results demonstrated that SOFM ANN achieved better performance than GA, where in SOFM ANN high attack detection rate is 99.98%, 99.89%, 100%, 100%, 100% and low false positive rate is 0.01%, 0.1%, 0%, 0%, 0% for DoS, R2L, Probe, U2R attacks, and Normal traffic respectively.

CLAPP: A self constructing feature clustering approach for anomaly detection

The term internet of things is a buzz word these days and as per Google survey conducted recently, it has even dominated the buzz word big data predominantly. However, IoT area is still not matured and is throwing light on lot of research issues towards the data mining researchers. Security in IoT throws several challenges because of limited resources. In this context, IoT gains importance once again from data miners towards anomaly mining or intrusion detection. Intrusion detection is classified as NP-class in the literature even today. Algorithms addressing privacy and security issues in IoT must consider the complexities involved and hence require re-attention from all researchers. One more problem faced when judging for intrusion is the use of high dimensionality, classifier choice, and distance measure. For example, the traditional distance measure, such as Euclidean misjudges the similarity. In this paper, the objective is to design a fuzzy membership function to address both dimensionality and anomaly mining so as reduce the computational complexity and increase computational accuracies of classifier algorithms. We validate the proposed measure using several experimentations on NSL-KDD and DARPA datasets using kNN, J48 and CANN using Gaussian measure. Improved accuracies of classifiers on U2R and R2L attacks have been recorded in the experimental results obtained for experiments conducted.

Intrusion detection in network flows based on an optimized clustering criterion

Graph-based intrusion detection approaches consider the network as a graph and detect anomalies based on graph metrics. However, most of these approaches succumb to the cluster-based behavior of the anomalies. To resolve this problem in our study, we use flow and graph-clustering concepts to create a data set first. A new criterion related to the average weight of clusters is then defined and a model is proposed to detect attacks based on the above-mentioned criterion. Finally, the model is evaluated using a DARPA data set. Results show that the proposed approach detects the attacks with high accuracy relative to methods described in previous studies.

Profile and Back Off Based Distributed NIDS in Cloud

Cloud security is a major concern these days. Out of the various attacks over Cloud the one which is very specific and consistently launched is DDoS. Current state of art solutions for detection of DDoS attacks in Cloud consume a lot of computational resources in performing per packet attack signature detection. As and when Cloud scales this will result in more resources being utilized for providing DDoS attack detection in Cloud eventually decreasing the amount of resources from the effective pool that can be allocated to its clients. We have utilized the underlying fact that during DDoS, attack packets are sent at a very heavy rate and hence proposed a profiling and back off based detection strategy for detecting DDoS attacks in Cloud. The solution provides lowest resource requirements at the same detection speed. The proposed solution is validated using DARPA dataset and has been thoroughly tested in multiple set of experimentations at client VM's in Cloud. It has provided a 100 % accuracy in DDoS attack detection with almost 32 times savior of computational resources at near to same detection speed compared to traditional per packet based NIDS at a back off detection value of T = 32.

A Hybrid Snort-Negative Selection Network Intrusion Detection Technique

Intrusion Detection Systems (NIDSs) are systems that monitor computer networks to detect, identify and prevent the malicious events, which attempt to compromise the integrity, confidentiality or availability of computer networks. The NIDS may be classified according to the detection technique into two types, the Signature-Based and Anomaly-Based NIDS. In order to increase the efficiency of the NIDS, a hybrid signature-anomaly NIDS based on both snort and negative selection algorithm is proposed. To evaluate the efficacy of the proposed system the 1999 DARPA data set is used. The experimental results show that the performance of the proposed system is more efficient than using snort on its own. General Terms

Real time intrusion detection system for ultra-high-speed big data environments

In recent years, the number of people using the Internet and network services is increasing day by day. On a daily basis, a large amount of data is generated over the Internet from zeta byte to petabytes with a very high speed. On the other hand, we see more security threats on the network, the Internet, websites, and the enterprise network. Therefore, detecting intrusion in such ultra-high-speed environment in real time is a challenging task. Many intrusion detection systems (IDSs) are proposed for various types of network attacks using machine learning approaches. Most of them are unable to detect recent unknown attacks, whereas the others do not provide a real-time solution to overcome the above-mentioned challenges. Therefore, to address these problems, we propose a real-time intrusion detection system for ultra-high-speed big data environment using Hadoop implementation. The proposed system includes four-layered IDS architecture, which consists of the capturing layer, filtration and load balancing layer, processing or Hadoop layer, and the decision-making layer. Furthermore, feature selection scheme is proposed that selects nine parameters for classification using (FSR) and (BER), as well as from the analysis of DARPA datasets. In addition, five major machine learning approaches are used to evaluate the proposed system including J48, REPTree, random forest tree, conjunctive rule, support vector machine, and Naive Bayes classifiers. Results show that among all these classifiers, REPTree and J48 are the best classifiers in terms of accuracy as well as efficiency. The proposed system architecture is evaluated with respect to accuracy in terms of true positive (TP) and false positive (FP), with respect to efficiency in terms of processing time and by comparing results with traditional techniques. It has more than 99 % TP and less than 0.001 % FP on REPTree and J48. The system has overall higher accuracy than existing IDSs with the capability to work in real time in ultra-high-speed big data environment.

FFSc: a novel measure for low‐rate and high‐rate DDoS attack detection using multivariate data analysis

AbstractA Distributed Denial of Service (DDoS) attack is a major security threat for networks and Internet services. Attackers can generate attack traffic similar to normal network traffic using sophisticated attacking tools. In such a situation, many intrusion detection systems fail to identify DDoS attack in real time. However, DDoS attack traffic behaves differently from legitimate network traffic in terms of traffic features. Statistical properties of various features can be analyzed to distinguish the attack traffic from legitimate traffic. In this paper, we introduce a statistical measure called Feature Feature score for multivariate data analysis to distinguish DDoS attack traffic from normal traffic. We extract three basic parameters of network traffic, namely, entropy of source IPs, variation of source IPs, and packet rate to analyze the behavior of network traffic for attack detection. The method is validated using CAIDA DDoS 2007 and MIT DARPA datasets. Copyright © 2016 John Wiley & Sons, Ltd.

Causality reasoning about network events for detecting stealthy malware activities

Malicious software activities have become more and more clandestine, making them challenging to detect. Existing security solutions rely heavily on the recognition of known code or behavior signatures, which are incapable of detecting new malware patterns. We propose to discover the triggering relations on network requests and leverage the structural information to identify stealthy malware activities that cannot be attributed to a legitimate cause. The triggering relation is defined as the temporal and causal relationship between two events. We design and compare rule- and learning-based methods to infer the triggering relations on network data. We further introduce a user-intention based security policy for pinpointing stealthy malware activities based on a triggering relation graph. We extensively evaluate our solution on a DARPA dataset and 7 GB real-world network traffic. Results indicate that our dependence analysis successfully detects various malware activities including spyware, data exfiltrating malware, and DNS bots on hosts. With good scalability for large datasets, the learning-based method achieves better classification accuracy than the rule-based one. The significance of our traffic reasoning approach is its ability to detect new and stealthy malware activities.

Improving Information Integrity using Artificial Bee Colony based Intrusion Detection System

Nowadays, Information, which is managed by Database Management System, is deemed as an asset for any of the organizations. Malicious attacks over a computer network can decrease security and trust of a system which may lead to various threats, which will be mitigated by introducing Intrusion Detection System. Emerging Intrusion detection systems (IDS) cannot improve the Integrity and in order to offer higher security for the confidential data with information integrity, an Artificial Bee Colony algorithm based Intrusion Detection system (ABC-IDS) is proposed. Clustering, Mining and Classification are the three major phases of this proposed system. The primary step is clustering the given datasets which helps to recover the quality of the datasets by partitioning the quantity in a cluster. Clustering is worked out by the clustering algorithm, Fuzzy C-Means and the second phase Mining is completed on the clustered datasets in order to get mined results by successfully mining the given datasets with the aid of Frequent Item set mining. The generated rules in the mining process get optimized by the usage of Artificial Bee Colony algorithm. After acquiring optimized mined results, the Classification phase is carried out by using Artificial Neural Network classifier, which classifies the input dataset into Intrusion or Non-Intrusion packets. This proposed method is implemented in MATLAB platform over DARPA dataset and then it is analyzed for its accuracy of Intrusion detection rate and Non-Intrusion detection rate, which will also evidently improve the consistency and reliability of the ABC-IDS system. Moreover, comparison of the proposed methodology with the state-of-art works is done to prove the improvement in information integrity in the proposed method. Hence, the optimum Integrity is increased by this research by the increase of its intrinsic attributes of accuracy, consistency and reliability.

Improved Intrusion Detection Technique based on Feature Reduction and Classification using Support Vector Machine and Particle of Swarm Optimization

Reduces the file size and increase the performances of classification and intrusion detection technique used in current research trend. The reduction of file size and number of attribute used dimension reduction algorithm and optimization algorithm. Various authors used genetic algorithm, ANT colony optimization and neural network. In this paper used particle of swarm optimization technique for feature reduction and feature selection for support vector machine classification process. The proposed algorithm implemented in MATLAB software and used DARPA dataset for evaluation of proposed method. Our empirical result shows that better detection ratio in compression of other exiting technique such as FCMNN, GSVM. Keyword IDS, Feature Reduction, SVM, PSO