Traffic Trace Data Research Articles

Neighbor Graph Based Tensor Recovery For Accurate Internet Anomaly Detection

Detecting anomalous traffic is a crucial task for network management. Although many anomaly detection algorithms have been proposed recently, constrained by their matrix-based traffic data model, existing algorithms often suffer from low detection accuracy. To fully utilize the multi-dimensional information hidden in the traffic data, this paper uses the tensor model for more accurate Internet anomaly detection. Only considering the low-rank linearity features hidden in the data, current tensor factorization techniques would result in low anomaly detection accuracy. We propose a novel Graph-based Tensor Recovery model (Graph-TR) to well explore both low-rank linearity features as well as the non-linear proximity information hidden in the traffic data for better anomaly detection. We encode the non-linear proximity information of the traffic data by constructing nearest neighbor graphs and incorporate this information into the tensor factorization using the graph Laplacian. Moreover, to facilitate the quick building of neighbor graph, we propose a nearest neighbor searching algorithm with the simple locality-sensitive hashing (LSH). Besides only detecting random anomalies, our algorithm can also effectively detect structured anomalies that appear as bursts. We have conducted extensive experiments using Internet traffic trace data Abilene and GÈANT. Compared with the state of art algorithms on matrix-based anomaly detection and tensor recovery approach, our Graph-TR can achieve higher Accuracy and Recall.

Calculation of AeroMACS Spectrum Requirements Based on Traffic Simulator.

In this paper, we propose a methodology for calculating the necessary spectrum requirements of aeronautical mobile airport communication system (AeroMACS) to provide various airport communication services. To accurately calculate the spectrum requirement, it is necessary to evaluate the AeroMACS traffic demand of the peak time and statistical data on the packet traffic generated at the airport. Because there is no AeroMACS traffic model and real trace data, we have developed the AeroMACS traffic simulator based on the report of Single European Sky Air Traffic Management Research (SESAR). To calculate the spectrum requirements, the AeroMACS traffic simulator is combined with the methodology of ITU-R M.1768-1. The developed traffic simulator reflects AeroMACS traffic priorities and can generate the required traffic according to its location in the airport. We observed the spectrum requirement by changing the number of sectors and the spectral efficiency. To show the feasibility of our methodology, we applied it to the case of Incheon International Airport in Korea. The simulation results show that the average bandwidth of 0.94 MHz is required in the ground area and 8.59 MHz is required in the entire airport.

SLRTA: A sparse and low-rank tensor-based approach to internet traffic anomaly detection

Internet traffic anomaly detection (ITAD) is a critical task for various network tasks such as traffic engineering and network security. Matrix-based approaches of ITAD have limitations for traffic data with multi-way structures, while the emerging tensor-based approaches of ITAD lack of sufficient consideration for circumstances including incomplete measurements or link-load measurements. To address these issues, we formulate ITAD by a sparse low-rank tensor optimization model, taking into full consideration the intrinsic and potential properties including the sparsity of anomalies, the low-rankness, the temporal stability and periodicity of the normal traffic data. Although the resulting optimization model is non-convex and discontinuous due to the involved ℓ0-norm and the tensor rank function, optimality analysis via stationarity is established, based on which an efficient proximal gradient method with theoretical convergence to stationary points is designed. Numerical experiments on internet traffic trace data Abilene and GÉANT demonstrate the high efficiency of our proposed sparse and low-rank tensor-based approach (SLRTA) for ITAD.

Accurate and Fast Recovery of Network Monitoring Data With GPU-Accelerated Tensor Completion

Monitoring the performance of a large network would involve a high measurement cost. To reduce the overhead, sparse network monitoring techniques may be applied to select paths or time intervals to take the measurements, while the remaining monitoring data can be inferred leveraging the spatial-temporal correlations among data. The quality of missing data recovery, however, highly relies on the specific inference technique adopted. Tensor completion is a promising technique for more accurate missing data inference by exploiting the multi-dimensional data structure. However, data processing for higher dimensional tensors involves a large amount of computation, which prevents conventional tensor completion algorithms from practical application in the presence of large amount of data. This work takes the initiative to investigate the potential and methodologies of performing parallel processing for high-speed and high accuracy tensor completion over Graphics Processing Units (GPUs). We propose a GPU-accelerated parallel Tensor Completion scheme (GPU-TC) for accurate and fast recovery of missing data. To improve the data recovery accuracy and speed, we propose three novel techniques to well exploit the tensor factorization structure and the GPU features: grid-based tensor partition, independent task assignment based on Fisher-Yates shuffle, sphere facilitated and memory-correlated scheduling. We have conducted extensive experiments using network traffic trace data to compare the proposed GPU-TC with the state of art tensor completion algorithms and matrix-based algorithms. The experimental results demonstrate that GPU-TC can achieve significantly better performance in terms of two relative error ratio metrics and computation time.

Vehicle-Based Bi-Objective Crowdsourcing

Mobile crowdsourcing is an emerging complex problem solving paradigm that makes use of pervasive mobile devices equipped with multi-functional sensors. Recently, vehicles have also been increasingly adopted for mobile crowdsourcing, as the vehicles, as well as drivers, can provide diverse sensing capability and predictable mobility. Existing mobile crowdsourcing algorithms mostly recruit workers to complete one kind of sensing tasks, i.e., location-based query tasks or automatic sensing tasks. In this paper, we investigate the possibility of recruiting a set of vehicles to simultaneously complete these two categories of tasks, so as to maximize the sensing utility of each participant. We first model the worker recruitment for vehicle-based crowdsourcing as a bi-objective optimization problem with respect to the sensing capability and predictable mobility of vehicles. The recruitment problem is proven to be NP-hard, and we design two heuristic algorithms based on the bi-objective greedy strategy and the multi-objective genetic algorithm to find the solutions. The experimental results with a real-world traffic trace data set show that the proposed algorithms outperform some existing algorithms in finding solutions that maximize both objectives.

On detecting compromised controller in software defined networks

While traditional networks depend on a fully distributed control plane, Software Defined Networks (SDNs), the rapidly emerging area in computer networking, utilize a centralized control plane. SDNs bring in many benefits such as fine-grained control, possibility of optimal routing, and resource management within the network. As a result, SDNs find wider deployments in certain segments of networking such as data center networks. In addition, SDN approach is a potential candidate for the control plane design in 5G networks. Despite the benefits, SDNs face certain issues such as the possibility of single point failure, the communication overhead between switches and controllers, and more importantly the security as well as trustability of the control plane. Due to the centralized nature of the control plane, it is important to detect the presence of compromised control plane in an SDN. Compromised control plane refers to the situation where one or more of the controllers in an SDN are compromised by malwares, resulting in deviation from the normal control plane behavior. Developing new solutions for detecting the presence of compromised controllers is exacerbated by the lack of appropriate SDN traffic data sets. As a result, existing literature lacks solutions to detect the presence of a compromised control plane. Of particular interest is the case where SDN controller-specific threats hide their presence from end-users and administrators of the network.Our contributions in this paper include the following: (i) identification of five threat vectors that represent compromised controllers in SDNs, (ii) creation of a large volume of OpenFlow traffic traces in order for studying various SDN threat vectors, (iii) proposal of nine novel OpenFlow-specific features that capture the above mentioned threat vectors, and (iv) study of machine-learning based detection technique for compromised control plane using six classifiers. The OpenFlow traffic trace data set, we created, is made available for the use of larger research community. We carried out detailed experimental studies that show the efficacy of our scheme in detecting the presence of compromised controllers. Our results indicate that Random Forest is the most suitable machine learning classifier that provides about 97% accuracy.

Fast Tensor Factorization for Accurate Internet Anomaly Detection

Detecting anomalous traffic is a critical task for advanced Internet management. Many anomaly detection algorithms have been proposed recently. However, constrained by their matrix-based traffic data model, existing algorithms often suffer from low accuracy in anomaly detection. To fully utilize the multi-dimensional information hidden in the traffic data, this paper takes the initiative to investigate the potential and methodologies of performing tensor factorization for more accurate Internet anomaly detection. More specifically, we model the traffic data as a three-way tensor and formulate the anomaly detection problem as a robust tensor recovery problem with the constraints on the rank of the tensor and the cardinality of the anomaly set. These constraints, however, make the problem extremely hard to solve. Rather than resorting to the convex relaxation at the cost of low detection performance, we propose TensorDet to solve the problem directly and efficiently. To improve the anomaly detection accuracy and tensor factorization speed, TensorDet exploits the factorization structure with two novel techniques, sequential tensor truncation and two-phase anomaly detection. We have conducted extensive experiments using Internet traffic trace data Abilene and GEANT. Compared with the state of art algorithms for tensor recovery and matrix-based anomaly detection, TensorDet can achieve significantly lower false positive rate and higher true positive rate. Particularly, benefiting from our well designed algorithm to reduce the computation cost of tensor factorization, the tensor factorization process in TensorDet is 5 (Abilene) and 13 (GEANT) times faster than that of the traditional Tucker decomposition solution.

Accurate traffic matrix completion based on multi-Gaussian models

Traffic matrix (TM) describes the volumes of traffic between a set of sources and destinations in a network. As an important parameter, TMs are used in a variety of network engineering tasks, such as traffic engineering, capacity planning and anomaly detection. However, it is a challenge to reliably measure TMs in practice. For example, due to flaws in the measurement systems and possible failure in data collection systems, missing values are unavoidable. It is important to recover the missing data from the partial direct measurements. Existing matrix completion methods do not fully consider network traffic behavior and traffic hidden characteristic. Their completion accuracy tends to be significantly worse when the data loss rate is high. In this paper, we perform a study on intrinsic characteristics of network traffic by analyzing real-world traffic trace data, which reveals that traffic has the features of temporal stability and spatial affinity. According to traffic spatial feature, we model TM as multi-Gaussian distributions, which describes the actual network traffic more accurately. Furthermore, we propose a novel matrix completion method based on multi-Gaussian models to estimate the missing traffic data. Finally, we utilize traffic temporal characteristic to further optimize traffic matrix completion for the missing data interpolation. Our proposed approach has been evaluated utilizing real-world traffic trace data. The extensive experiments demonstrate that our method achieves significantly better performance compared with the state-of-the-art interpolation methods.

Entropy and Flow-based Approach for Anomalous Traffic Filtering

Research Tools in Anomaly-based Intrusion Detection are highly dependent on appropriate traffic trace data. Traditional datasets present several issues such as: removal of sensitive information (anonymization) and insufficient number or volume of attack instances, which limit their quality for the design and evaluation of A-NIDSs. In this paper, we present a method for anomalous traffic filtering which can be used for generating anomaly-free traffic traces. The sanitized dataset can be used to improve the computation of the behaviour profiles during the training stage. The proposal is based on the construction and statistical analysis of the flow-level entropy space for the identification of outliers using three entropy estimators. Empirical results showed that the new traffic traces of the sanitized dataset have a distributional similarity among them greater than that presented among the original datasets.

Multiplicative multifractal modelling of long‐range‐dependent network traffic

AbstractWe present a multiplicative multifractal process to model traffic which exhibits long‐range dependence. Using traffic trace data captured by Bellcore from operations across local and wide area networks, we examine the interarrival time series and the packet length sequences. We also model the frame size sequences of VBR video traffic process. We prove a number of properties of multiplicative multifractal processes that are most relevant to their use as traffic models. In particular, we show these processes to characterize effectively the long‐range dependence properties of the measured processes. Furthermore, we consider a single server queueing system which is loaded, on one hand, by the measured processes, and, on the other hand, by our multifractal processes (the latter forming a MFe/MFg/1 queueing system model). In comparing the performance of both systems, we demonstrate our models to effectively track the behaviour exhibited by the system driven by the actual traffic processes. We show the multiplicative multifractal process to be easy to construct. Through parametric dependence on one or two parameters, this model can be calibrated to fit the measured data. We also show that in simulating the packet loss probability, our multifractal traffic model provides a better fit than that obtained by using a fractional Brownian motion model. Copyright © 2001 John Wiley & Sons, Ltd.