Traffic Anomaly Detection Research Articles

ENTVis: A Visual Analytic Tool for Entropy-Based Network Traffic Anomaly Detection.

Entropy-based traffic metrics have received substantial attention in network traffic anomaly detection because entropy can provide fine-grained metrics of traffic distribution characteristics. However, some practical issues--such as ambiguity, lack of detailed distribution information, and a large number of false positives--affect the application of entropy-based traffic anomaly detection. In this work, we introduce a visual analytic tool called ENTVis to help users understand entropy-based traffic metrics and achieve accurate traffic anomaly detection. ENTVis provides three coordinated views and rich interactions to support a coherent visual analysis on multiple perspectives: the timeline group view for perceiving situations and finding hints of anomalies, the Radviz view for clustering similar anomalies in a period, and the matrix view for understanding traffic distributions and diagnosing anomalies in detail. Several case studies have been performed to verify the usability and effectiveness of our method. A further evaluation was conducted via expert review.

A hybrid local and distributed sketching design for accurate and scalable heavy key detection in network data streams

Real-time characterization of network traffic anomalies, such as heavy hitters and heavy changers, is critical for the robustness of operational networks, but its accuracy and scalability are challenged by the ever-increasing volume and diversity of network traffic. We address this problem by leveraging parallelization. We propose LD-Sketch, a data structure designed for accurate and scalable traffic anomaly detection using distributed architectures. LD-Sketch combines the classical counter-based and sketch-based techniques, and performs detection in two phases: local detection, which guarantees zero false negatives, and distributed detection, which reduces false positives by aggregating multiple detection results. We derive the error bounds and the space and time complexity for LD-Sketch. We further analyze the impact of ordering of data items on the memory usage and accuracy of LD-Sketch. We compare LD-Sketch with state-of-the-art sketch-based techniques by conducting experiments on traffic traces from a real-life 3G cellular data network. Our results demonstrate the accuracy and scalability of LD-Sketch over prior approaches.

NETWORK TRAFFIC ANOMALIES IDENTIFICATION BASED ON CLASSIFICATION METHODS / TINKLO SRAUTO ANOMALIJŲ IDENTIFIKAVIMAS, TAIKANT KLASIFIKAVIMO METODUS

A problem of network traffic anomalies detection in the computer networks is analyzed. Overview of anomalies detection methods is given then advantages and disadvantages of the different methods are analyzed. Model for the traffic anomalies detection was developed based on IBM SPSS Modeler and is used to analyze SNMP data of the router. Investigation of the traffic anomalies was done using three classification methods and different sets of the learning data. Based on the results of investigation it was determined that C5.1 decision tree method has the largest accuracy and performance and can be successfully used for identification of the network traffic anomalies. Straipsnyje nagrinėjama kompiuterių tinklo srauto anomalijų atpažinimo problema. Apžvelgiami kompiuterių tinklų anomalijų aptikimo metodai bei aptariami jų privalumai ir trūkumai. Naudojant IBM SPSS Modeler programų paketą sudarytas nagrinėjamo tinklo srauto anomalijų atpažinimo modelis, pritaikytas SNMP protokolu pagrįstiems maršruto parinktuvo duomenims apdoroti. Pagal tris klasifikavimo metodus ir skirtingus mokymui skirtus duomenų rinkinius atlikti skaičiavimai tinklo anomalijoms identifikuoti. Palyginant gautus rezultatus nustatyta, kad C5.1 sprendimo medžio algoritmas yra tiksliausias ir sparčiausias, todėl ir tinkamiausias tinklo srauto anomalijoms atpažinti.

Statistical Traffic Anomaly Detection in Time-Varying Communication Networks

We propose two methods for traffic anomaly detection in communication networks where properties of normal traffic evolve dynamically. We formulate the anomaly detection problem as a binary composite hypothesis testing problem and develop a model-free and a model-based method, leveraging techniques from the theory of large deviations. Both methods first extract a family of probability laws (PLs) that represent normal traffic patterns during different time-periods, and then detect anomalies by assessing deviations of traffic from these laws. We establish the asymptotic Newman-Pearson optimality of both methods and develop an optimization-based approach for selecting the family of PLs from past traffic data. We validate our methods on networks with two representative time-varying traffic patterns and one common anomaly related to data exfiltration. Simulation results show that our methods perform better than their vanilla counterparts, which assume that normal traffic is stationary.

Traffic anomaly detection based on image descriptor in videos

The huge and ever growing volume of traffic video poses a compelling demand for efficient automatic detection of traffic anomaly. In this paper, a new traffic anomaly detection algorithm is introduced. It firstly divides a traffic video into several video cubes in temporal domain, and each video cube is divided into video blocks in spatial domain. Each image block of a video block is described using the local invariant features and the visual codebook approach. Based on the descriptor of the image block, we count the category number of the block (CNB) of a video block. Then, a Gaussian distribution model for estimating the probability of normal traffic with respect to the CNB is learned. The learned Gaussian distribution model is then used to detect the traffic anomaly from the test traffic video. Eventually, the results of all video blocks are fused to achieve the final decision. Experimental results show that the proposed algorithm performs better than two existing algorithms on both the intersection traffic videos and main road traffic videos.

Defining Generic Attributes for IDS Classification

Detection accuracy of Intrusion Detection System (IDS) depends on classifying network traffic based on data features. Using all features for classification consumes more computation time and computer resources. Some of these features may be redundant and irrelevant therefore, they affect the detection of traffic anomalies and the overall performance of the IDS. The literature proposed different algorithms and techniques to define the most relevant sets of features of KDD cup 1999 that can achieve high detection accuracy and maintain the same performance as the total data features. However, all these algorithms and techniques did not produce optimal solutions even when they utilized same datasets. In this paper, a new approach is proposed to analyze the researches that have been conducted on KDD cup 1999 for features selection to define the possibility of determining effective generic features of the common dataset KDD cup 1999 for constructing an efficient classification model. The approach does not rely on algorithms, which shortens the computational cost and reduces the computer resources. The essence of the approach is based on selecting the most frequent features of each class and all classes in all researches, then a threshold is used to define the most significant generic features. The results revealed two sets of features containing 7 and 8 features. The classification accuracy by using eight features is almost the same as using all dataset features.

Network-Wide Traffic Anomaly Detection and Localization Based on Robust Multivariate Probabilistic Calibration Model

Network anomaly detection and localization are of great significance to network security. Compared with the traditional methods of host computer, single link and single path, the network-wide anomaly detection approaches have distinctive advantages with respect to detection precision and range. However, when facing the actual problems of noise interference or data loss, the network-wide anomaly detection approaches also suffer significant performance reduction or may even become unavailable. Besides, researches on anomaly localization are rare. In order to solve the mentioned problems, this paper presents a robust multivariate probabilistic calibration model for network-wide anomaly detection and localization. It applies the latent variable probability theory with multivariatet-distribution to establish the normal traffic model. Not only does the algorithm implement network anomaly detection by judging whether the sample’s Mahalanobis distance exceeds the threshold, but also it locates anomalies by contribution analysis. Both theoretical analysis and experimental results demonstrate its robustness and wider use. The algorithm is applicable when dealing with both data integrity and loss. It also has a stronger resistance over noise interference and lower sensitivity to the change of parameters, all of which indicate its performance stability.

NETWORK TRAFFIC ANOMALIES DETECTING USING MAXIMUM ENTROPY METHOD / KOMPIUTERIŲ TINKLO SRAUTO ANOMALIJŲ ATPAŽINIMAS MAKSIMALIOS ENTROPIJOS METODU

The problem of traffic anomalies in computer networks is analyzed. NetFlow packets are used as network traffic data and maximum entropy methods is used for anomalies detection. Method detects network anomalies by comparing the current network traffic against a baseline distribution. Method is adopted according to NetFow data and performace of the method is improved. Prototype of anomalies detection system was developed and experimental investigation carried out. Results of investigation confirmed that method is sensitive to deviations of the network traffic and can be successfully used for network traffic anomalies detection. Straipsnyje nagrinėjama kompiuterių tinklo srauto anomalijų atpažinimo problema. Kompiuterių tinklo srautui stebėti pasirenkama NetFlow technologija, o anomalijos aptinkamos maksimalios entropijos metodu. Metodas pritaikytas NetFlow pateikiamiems duomenims apdoroti. Sukurta programinė priemonė ir atliktas eksperimentinis metodo tinkamumo tyrimas analizuojant „Cisco“ maršrutizatoriaus srauto duomenis. Metodas patobulintas siekiant pagreitinti skaičiavimus, tačiau neprarandant tikslumo. Nustatyta, kad metodas yra jautrus įvairaus tipo tinklo srauto nuokrypiams ir gali būti sėkmingai taikomas tinklo srauto anomalijoms aptikti.

Network Traffic Anomalies, Natural Language Processing, and Random Matrix Theory

Random Matrix Theory (RMT) is an important tool for detecting correlations in multidimensional time series, such as stock market price histories, and origin-destination flows in data networks.We review the basic theory and propose two novel applications: the detection of traffic anomalies in data networks and natural language processing.For traffic anomalies the advantage of this approach is that training sets are not necessary. In the case of natural language processing, our approach is a refinement of the standard Latent Semantic Analysis (LSA).We will demonstrate applications to real traffic from a data network, and present the use in Natural Language Processing. Directions for future work will be discussed.

A Hybrid Technique Using PCA and Wavelets in Network Traffic Anomaly Detection

Research into network anomaly detection has become crucial as a result of a significant increase in the number of computer attacks. Many approaches in network anomaly detection have been reported in the literature, but data or solutions typically are not freely available. Recently, a labeled network traffic flow dataset, Kyoto2006+, has been created and is publicly available. Most existing approaches using Kyoto2006+ for network anomaly detection apply various clustering techniques. This paper leverages existing well known statistical analysis and spectral analysis techniques for network anomaly detection. The first popular approach is a statistical analysis technique called Principal Component Analysis (PCA). PCA describes data in a new dimension to unlock otherwise hidden characteristics. The other well known spectral analysis technique is Haar Wavelet filtering analysis. It measures the amount and magnitude of abrupt changes in data. Both approaches have strengths and limitations. In response, this paper proposes a Hybrid PCA–Haar Wavelet Analysis. The hybrid approach first applies PCA to describe the data and then Haar Wavelet filtering for analysis. Based on prototyping and measurement, an investigation of the Hybrid PCA–Haar Wavelet Analysis technique is performed using the Kyoto2006+ dataset. The authors consider a number of parameters and present experimental results to demonstrate the effectiveness of the hybrid approach as compared to the two algorithms individually.

Anomaly diagnosis based on regression and classification analysis of statistical traffic features

ABSTRACTTraffic anomalies caused by Distributed Denial‐of‐Service (DDoS) attacks are major threats to both network service providers and legitimate customers. The DDoS attacks regularly consume and exhaust the resources of victims and hence result in abnormal bursty traffic through end‐user systems. Additionally, malicious traffic aggregated into normal traffic often show dramatic changes in the traffic nature and statistical features. This study focuses on early detection of traffic anomalies caused by DDoS attacks in light of analyzing the network traffic behavior. Key statistical features including variance, autocorrelation, and self‐similarity are employed to characterize the network traffic. Further, artificial neural network and support vector machine subject to the performance metrics are employed to predict and classify the abnormal traffic. The proposed diagnosis mechanism is validated through experiments where the datasets consist of two groups. The first group is the Massachusetts Institute of Technology Lincoln Laboratory dataset containing labeled DoS attack. The second group collected from DDoS attack simulation experiments covers three representative traffic shapes resulting from the dynamic attack rate configuration, namely, constant intensity, ramp‐up behavior, and pulsing behavior. The experimental results demonstrate that the developed mechanism can effectively and precisely alert the abnormal traffic within short response period. Copyright © 2013 John Wiley & Sons, Ltd.

Traffic Flow Anomaly Detection Based on Wavelet Denoising and Support Vector Regression

In order to improve the speed and accuracy of traffic flow anomaly detection in real-time traffic system, we proposed an anomaly detection algorithm which is based on wavelet denoising and support vector regression. Firstly, we use wavelet transform to decompose and restructure the sampled data, and then apply support vector regression to data training. By fitting the obtained data, it can achieve dynamic prediction of traffic flow parameters. Through comparing the predictive values with the measured values of traffic flow parameters, we can achieve traffic anomaly detection. Experimental results show that the method proposed in this paper has a higher detection rate under the same false alarm rate.

How to Extract and Train the Classifier in Traffic Anomaly Detection System

With the continuous development of network security,various types of traffic anomaly detection systems have been proposed and the system based on classification is one of the most important classes.However,because the network environment is diversity and dynamic changing,the traffic anomaly detection systems confront with high false positive rate when it is deployed who can detect every anomaly in the training dataset exactly.It is difficult to extract precision classifier and classifier should adapt with time and environment.In order to overcome these problems mentioned above,algorithms and data struts about how to extract and train the classifier are proposed.The network traffic is projected to different Hash histograms over different dimensions,and these histograms,through a support vector data description machine,are classified into two groups,abnormal and normal;secondly,in order to improve the accuracy of detection and reduce training time,the classifier is trained continuously to describe the real-time and dynamic network traffic exactly.Third,the multi-windows correlation algorithm is adopted to calculate a comprehensive anomaly score through a sequence of windows and weed out obvious outliers in the new training set.Using some real and synthetic traffic traces from backbone networks in the Internet,we demonstrate that our system can detect anomalies with higher accuracy,lower computation and memory costs than other systems which were widely used in anomaly detection.

Traffic Anomaly Detection Improvement Based on Spatial-Temporal Characteristics

Traffic Anomaly Detection Improvement Based on Spatial-Temporal Characteristics

An Online Learning and Unsupervised Traffic Anomaly Detection System

An Online Learning and Unsupervised Traffic Anomaly Detection System

Designing an Approach for Network Traffic Anomaly Detection

traffic for anomaly detection. The accurate and rapid detection of network traffic anomaly is crucial to enhance the effective operation of a network. It is often difficult to detect the time when the faults occur in a network. In this paper, a new algorithm is presented to monitor the aggregate network traffic to rapidly detect the time anomaly occurs in a network. This is accomplished by monitoring the statistical characteristics of the time series representing the network behavior. The technique analyzes the network behavior using fractal dimension and discrete stationary wavelet transform. In the proposed method, after applying discrete stationary wavelet transform on the signal representing the network traffic, the fractal dimension of the decomposed signal is calculated in a sliding window. Then, variations of signal fractal dimension are considered for anomaly detection. Performance of the proposed method is compared with that of three other existing methods using both synthetic signal and real data. The results indicate superiority of the proposed technique in terms of accuracy compared to existing methods.

Traffic Anomaly Detection and Containment Using Filter-Ary-Sketch

Traffic anomaly detection systems require not only efficiency and accuracy but also the ability of containment. For those special requirements, an anomaly detection system is proposed based on Filter-ary-Sketch. It records the traffic in Filter-ary-Sketch and detects anomalies over it. When an anomaly is detected, the anomalous dimension is notices and malicious buckets are identified. Finally, malicious packets are blocked using the packets filter algorithm. Using some traffic traces from backbone networks, we demonstrate that our system can detect anomalies with high accuracy, low computation and memory costs, and block the packets that are responsible for anomalies.

Detection and Classification of Traffic Anomalies Using Microscopic Traffic Variables

This paper proposes a novel anomaly detection and classification algorithm that combines the spatiotemporal changes in the variability of microscopic traffic variables, namely, relative speed, intervehicle time gap, and lane changing. When applied to real-world scenarios, the proposed algorithm can use the variances of statistics of microscopic traffic variables to detect and classify traffic anomalies. Based on a simulation environment, it is shown that, with minimum prior knowledge and partial availability of microscopic traffic information from as few as 20% of the vehicle population, the proposed algorithm can still achieve 100% detection rates and very low false alarm rates, which outperforms previous algorithms monitoring loop detectors that are ideally placed at locations where anomalies originate.

A Novel Network Traffic Anomaly Detection Based on Multi-Scale Fusion

Detecting network traffic anomaly is very important for network security. But it has high false alarm rate, low detect rate and that can’t perform real-time detection in the backbone very well due to its nonlinearity, nonstationarity and self-similarity. Therefore we propose a novel detection method—EMD-DS, and prove that it can reduce mean error rate of anomaly detection efficiently after EMD. On the KDD CUP 1999 intrusion detection evaluation data set, this detector detects 85.1% attacks at low false alarm rate which is better than some other systems.

Network Traffic Anomaly Detection Method Based on a Feature of Catastrophe Theory

For the existing problems of current network traffic anomaly detection, the behavior of the network traffic anomaly will show nonlinearity, non-stationarity and complexity according to the network traffic often driven by the control of multiple factors. Owing to the characteristic that the internal evolution equation will lead to dynamical structure catastrophe, the phase space reconstruction method and the statistical physics method can be used to compute the macro feature values of the network traffic. By choosing some of the feature values which can obviously reflect the unusual change in the network traffic volume as control variables, a network traffic anomaly detection method based on the catastrophe series theory model is developed. Many experimental results show that the proposed network traffic anomaly detection method has a low false alarm rate under the same condition of detection rate.