Network Traffic Measurements Research Articles

Server load estimation by Burr distribution mixture analysis of TCP SYN response time

Server load estimation is key in balancing traffic between servers when optimizing data center resources. Intrusive methods are sometimes difficult or impossible to implement. Therefore, non-intrusive estimation methods are the best alternative in these cases. The objective of this paper is to present a server load estimation method based on external network traffic measurements obtained in a vantage point close to the server. Statistical distributions of TCP SYN response time, that is, the time from SYN to SYN+ACK segments at the server side, are used to fit Burr Type XII heavy tail distribution mixtures. The fitting algorithm, based on maximum likelihood estimation, is developed in detail in this paper. Experimental data shows that the median of the fitted distribution correlates within the 95% confidence interval of the server load figures and, thus, it can be used as a non-intrusive and accurate method to measure it. This new method can be applied to almost any existing load balancing algorithm, as it does not make any assumption about the server, which is considered a black box.

Simulation-Based Evacuation System for Large-Scale Urban Road Network Considering Traffic Conditions

In case of an emergency, it is necessary to consider the impact of all traffic conditions on implementing the evacuation program accurately and reliably because there are still some private vehicles running in the evacuation area. To tackle this issue, this paper presents a large-scale simulation-based evacuation system from multiple evacuation depots to multiple safety depots considering traffic conditions, including three parts: traffic conditions acquisition, evacuation path planning, and microsimulation evaluation. Firstly, the traffic conditions related to link travel time between two points are estimated by considering private vehicles running on the road network and the rescue bus vehicles participating in the evacuation. A multi-path choice model for a dynamic time-varying network is then proposed, which might solve the evacuation paths with the minimum of evacuation time. Experimental results show that the proposed method could have higher accuracy than the traditional static path planning, as well as higher computational efficiency than the dynamic algorithm. Finally, the variable cell transmission model (VCTM) is improved to deduce the characteristics of evacuation traffic flow under the estimated traffic conditions, which can be used to recognize the bottlenecks in the road network and re-optimize traffic control measurements. The contrastive analysis results showed that the improved VCTM is in good agreement with the operation results of the popular microscopic simulation software VISSIM.

Application of Utility Mining using Frequent Itemset and Association Rules: A Survey

Mining on data reveals patterns that provide useful information for analysis, decision making and forecasting in various domains. Association Rule Mining (ARM) identifies patterns on itemsets which are either frequent or have interesting relationship amongst them based on strong rules and conceptually form a basis for Frequent Itemset mining (FIM) problems. FIM extracts binary values from transaction databases to identify frequently bought items but provides insufficient information for identifying infrequent items that generate maximum profit. So a latter problem, High utility itemsets (HUI) mining was developed to focus on the itemsets that generate huge profit to the business. Even though HUI is related to Business Intelligence, its application extends to Web Server Logs, Biological Gene Databases, Network Traffic Measurements and many other fields. This paper presents a survey on the algorithms from different aspects and perspectives based on Utility mining, Frequent Itemset generation and Association Rule Mining

A New Traffic Prediction Algorithm to Software Defined Networking

Traffic prediction is significantly important for performance analysis and network planning in Software Defined Networking (SDN). However, to effectively predict network traffic in current networks is very difficult and nearly prohibitive. As a new cutting-edge network technology, SDN decouples the control and data planes of network switch devices to enable the flexibility of network measurements and managements. The SDN architecture of the flow-based forwarding idea brings forth a promising of network traffic capture and prediction. We propose a lightweight traffic prediction algorithm for SDN applications. Firstly, different from traditional network traffic measurements, our method uses the flow-based forwarding idea in SDN to extract traffic statistic from data plane. The statistical variable describes network flow information forwarded in SDN and enables more accurate measurements of flow traffic via a direct and low-overhead way compared with traditional traffic measurements. Secondly, based on the temporal nature of traffic, the time-correlation theory is utilized to model flow traffic, where the time-series analysis theory and regressive modeling approach are used to characterize network traffic in SDN. A fully new method is proposed to perform traffic prediction. Thirdly, we propose the flow-based forwarding traffic prediction algorithm to forecast to SDN traffic. The detailed algorithm process is discussed and analyze. Finally, sufficient experiments are presented and designed to validate the proposed method. Simulation results show that our method is feasible and effective.

DeepSec meets RawPower - Deep Learning for Detection of Network Attacks Using Raw Representations

The application of machine learning models to the analysis of network traffic measurements has largely grown in recent years. In the networking domain, shallow models are usually applied, where a set of expert handcrafted features are needed to fix the data before training. There are two main problems associated with this approach: firstly, it requires expert domain knowledge to select the input features, and secondly, different sets of custom-made input features are generally needed according to the specific target (e.g., network security, anomaly detection, traffic classification). On the other hand, the power of machine learning models using deep architectures (i.e., deep learning) for networking has not been yet highly explored. In this paper we explore the power of deep learning models on the specific problem of detection of network attacks, using different representations for the input data. As a mayor advantage as compared to the state of the art, we consider raw measurements coming directly from the stream of monitored bytes as the input to the proposed models, and evaluate different raw-traffic feature representations, including packet and flow-level ones.

IoV distributed architecture for real-time traffic data analytics

In this paper, we present necessary premises for the deployment of the Internet of Vehicles (IoV) integrating Big Data analytics of road network traffic measurements of the city of Mohammedia, Morocco. Thus, we introduce an architecture based on three main layers such as IoV, Fog Computing and Cloud Computing Layer. We specifically put more focus on Fog Computing layer in which we develop a framework for a real-time collecting and processing events generated by intelligent vehicles as well as visualizing traffic state on each road section. Furthermore, we consider deployment and test of the proposed framework using events retrieved from a Vanets-type micro simulation. Finally, we present and discuss the first obtained results as well as the advantages and limitations of the proposed architecture.

Scalability Issues for Remote Sensing Infrastructure: A Case Study.

For the past decade, a team of University of Calgary researchers has operated a large “sensor Web” to collect, analyze, and share scientific data from remote measurement instruments across northern Canada. This sensor Web receives real-time data streams from over a thousand Internet-connected sensors, with a particular emphasis on environmental data (e.g., space weather, auroral phenomena, atmospheric imaging). Through research collaborations, we had the opportunity to evaluate the performance and scalability of their remote sensing infrastructure. This article reports the lessons learned from our study, which considered both data collection and data dissemination aspects of their system. On the data collection front, we used benchmarking techniques to identify and fix a performance bottleneck in the system’s memory management for TCP data streams, while also improving system efficiency on multi-core architectures. On the data dissemination front, we used passive and active network traffic measurements to identify and reduce excessive network traffic from the Web robots and JavaScript techniques used for data sharing. While our results are from one specific sensor Web system, the lessons learned may apply to other scientific Web sites with remote sensing infrastructure.

Traffic Measurements for Cyber Security

The articles in this special section focus on telecommunication traffic measurements for protecting network security. Computers and open communication networks have become increasingly interwoven with our daily lives and have profoundly changed our societies. While this has significantly increased people’s well being, our growing dependence on an increasingly pervasive, complex, and ever evolving network infrastructure also poses a wide range of cyber security risks with potentially large socio-economic impacts. For example, the increasing number of ill-secured networked devices in combination with growing network capacities enables miscreants to launch disruptive distributed denial of service (DDoS) attacks. Within this context, network traffic measurements and monitoring have become a crucial line of research. It enables us to enhance our understanding of cyber security threats and use this knowledge to develop new ways to detect and mitigate them. Example applications of network measurement research include the analysis of how malicious software proliferates and operates, and how it exploits users’ behavior, assessments of the effectiveness of cyber security countermeasures, of the “badness” of Internet service providers, and estimations of the revenues of cyber criminals.

SeLINA: A Self-Learning Insightful Network Analyzer

Understanding the behavior of a network from a large scale traffic dataset is a challenging problem. Big data frameworks offer scalable algorithms to extract information from raw data, but often require a sophisticated fine-tuning and a detailed knowledge of machine learning algorithms. To streamline this process, we propose self-learning insightful network analyzer (SeLINA), a generic, self-tuning, simple tool to extract knowledge from network traffic measurements. SeLINA includes different data analytics techniques providing self-learning capabilities to state-of-the-art scalable approaches, jointly with parameter auto-selection to off-load the network expert from parameter tuning. We combine both unsupervised and supervised approaches to mine data with a scalable approach. SeLINA embeds mechanisms to check if the new data fits the model, to detect possible changes in the traffic, and to, possibly automatically, trigger model rebuilding. The result is a system that offers human-readable models of the data with minimal user intervention, supporting domain experts in extracting actionable knowledge and highlighting possibly meaningful interpretations. SeLINA’s current implementation runs on Apache Spark. We tested it on large collections of real-world passive network measurements from a nationwide ISP, investigating YouTube, and P2P traffic. The experimental results confirmed the ability of SeLINA to provide insights and detect changes in the data that suggest further analyses.

Errata for: Estimating internet address space usage through passive measurements (SIGCOMM CCR (Vol. 44, Issue 1, January, 2014)

One challenge in understanding the evolution of Internet in- frastructure is the lack of systematic mechanisms for monitoring the extent to which allocated IP addresses are actually used. Address utilization has been monitored via actively scanning the entire IPv4 address space. We evaluate the potential to leverage passive network traffic measurements in addition to or instead of active probing. Passive traffic measurements introduce no network traffic overhead, do not rely on unfiltered responses to probing, and could potentially apply to IPv6 as well. We investigate two challenges in using passive traffic for address utilization inference: the limited visibility of a single observation point; and the presence of spoofed IP addresses in packets that can distort results by implying faked addresses are active. We propose a methodology for removing such spoofed traf- fic on both darknets and live networks, which yields results comparable to inferences made from active probing. Our preliminary analysis reveals a number of promising findings, including novel insight into the usage of the IPv4 address space that would expand with additional vantage points.

Streaming Solutions for Fine-Grained Network Traffic Measurements and Analysis

Online network traffic measurements and analysis is critical for detecting and preventing any real-time anomalies in the network. We propose, implement, and evaluate an online, adaptive measurement platform, which utilizes real-time traffic analysis results to refine subsequent traffic measurements. Central to our solution is the concept of Multi-Resolution Tiling (MRT), a heuristic approach that performs sequential analysis of traffic data to zoom into traffic subregions of interest. However, MRT is sensitive to transient traffic spikes. In this paper, we propose three novel traffic streaming algorithms that overcome the limitations of MRT and can cater to varying degrees of computational and storage budgets, detection latency, and accuracy of query response. We evaluate our streaming algorithms on a highly parallel and programmable hardware as well as a traditional software-based platforms. The algorithms demonstrate significant accuracy improvement over MRT in detecting anomalies consisting of synthetic hard-to-track elephant flows and global icebergs. Our proposed algorithms maintain the worst-case complexities of the MRT while incurring only a moderate increase in average resource utilization.

Estimating internet address space usage through passive measurements

One challenge in understanding the evolution of Internet infrastructure is the lack of systematic mechanisms for monitoring the extent to which allocated IP addresses are actually used. Address utilization has been monitored via actively scanning the entire IPv4 address space. We evaluate the potential to leverage passive network traffic measurements in addition to or instead of active probing. Passive traffic measurements introduce no network traffic overhead, do not rely on unfiltered responses to probing, and could potentially apply to IPv6 as well. We investigate two challenges in using passive traffic for address utilization inference: the limited visibility of a single observation point; and the presence of spoofed IP addresses in packets that can distort results by implying faked addresses are active. We propose a methodology for removing such spoofed traffic on both darknets and live networks, which yields results comparable to inferences made from active probing. Our preliminary analysis reveals a number of promising findings, including novel insight into the usage of the IPv4 address space that would expand with additional vantage points.

An Analysis on End-To-End Inference Methods based On Packet Probing in Network

Current Internet is a massive, distributed network which continues to grow in size as globalization takes major role in everyone’s life like e-commerce, social networking and related activities grow. The heterogeneous and largely unregulated structure of the Internet renders tasks such as optimized service provision, rate limiting certain classes of applications (e.g. peer-to-peer), provide bandwidth guarantee for certain applications, avoiding shared congestion in flows are increasingly challenging tasks. The problem is complicated by the fact that one cannot rely on the cooperation of individual servers and routers to aid in the collection of network traffic measurements vital for these tasks. Hence we go for network monitoring and inference methods based on packet probing in the network. This paper presents an analysis of different inference methods for network characteristics to deal with shared congestion, packet forwarding priority, network tomography and evaluates each methodology based on packet loss rate and delay variance.

Internet Traffic Prediction Using Artificial Neural Network

In this paper we employed neural network on real network traffic measurements to predict Internet traffic. The traffic data consists of hourly traces from freestats.com collected over periods of 7 and 82 days respectively. First, self-similarity tests were carried out on the data series to ascertain its self-similar nature. The data were aggregated at various frequencies to smooth out burstiness. A feed-forward multi-layer perceptions (MLP) neural network was constructed and trained to predict the traffic. The results show that, using neural network, a reliable prediction can be made on packet data despite its self-similar profile. The results also showed that a neural network trained on a self-similar series can reliably predict the aggregated forms of the series. It was concluded that the results obtained can be used to moderate congestion using limited network resources and sharing the bandwidth allocation. Keywords: Self-similarity, neural networks, Hurst parameter, Internet traffic, aggregation, long range dependence (LRD).

Periodicity-Based Anomalies in Self-Similar Network Traffic Flow Measurements

Network traffic flow measurement is fundamental in timely monitoring computer networks and in diagnosing potential anomalies. Previous measurement studies have shown that network traffic flows are often self-similar. The degree of self-similarity is described by the Hurst parameter H. In the literature, various methods have been used in estimating H, while their performances have not been evaluated for network traces that contain periodicity-based anomalies. In this paper, we investigate the performance of well-known estimators for traffic flow measurements with periodicity-based anomalies. We derive analytical expressions for widely used estimation methods in time, frequency, wavelet, and eigen domains and demonstrate through simulations that periodicity-based anomalies affect Hurst parameter estimation, causing unreliable H estimates. We show that our theoretical and experimental results are consistent with the observations of real network traffic flow measurements.

다중화된 네트워크 트래픽의 self-similar 특성 분석에 관한 연구

In this paper, self-similar characteristics over statistical approaches and real-time Ethernet network traffic measurements are estimated. It is also shown that the self-similar traffic reflects real Ethernet traffic chareacteristics by comparing TCP-MT source model which is exactly self-similar model to the traditional Poisson model.

Fast, Memory Efficient Flow Rate Estimation Using Runs

Per-flow network traffic measurements are needed for effective network traffic management, network performance assessment, and detection of anomalous network events such as incipient denial-of-service (DoS) attacks. Explicit measurement of per-flow traffic statistics is difficult in backbone networks because tracking the possibly hundreds of thousands of flows needs correspondingly large high-speed memories. To reduce the measurement overhead, many previous papers have proposed the use of random sampling and this is also used in commercial routers (Cisco's NetFlow). Our goal is to develop a new scheme that has very low memory requirements and has quick convergence to within a pre-specified accuracy. We achieve this by use of a novel approach based on sampling two-runs to estimate per-flow traffic. (A flow has a two-run when two consecutive samples belong to the same flow). Sampling two-runs automatically biases the samples towards the larger flows thereby making the estimation of these sources more accurate. This biased sampling leads to significantly smaller memory requirement compared to random sampling schemes. The scheme is very simple to implement and performs extremely well.

Remote analysis of a distributed WLAN using passive wireless-side measurement

This paper presents network traffic measurements from a campus-wide wireless LAN (WLAN), with the data collected using remote passive wireless-side measurement. We used commercially-available monitoring devices to collect wireless traffic concurrently from 9 selected locations on the campus WLAN for 6 weeks. The aggregate trace contains almost 1 billion wireless frames, representing the WLAN activity generated by 6775 users and 97 access points. Analysis of the dataset identifies similarities and differences in the user behaviours across the observed WLAN locations, as well as emerging trends in WLAN usage regarding application usage and session mobility. Our study extends existing WLAN measurement studies by providing deeper insights into how WLANs are used, and by developing models of WLAN usage characteristics that are applicable in capacity planning, network testing, and network simulation studies.

Methodology To Assess the Quality of Service Provision

This paper presents a new methodology to assess the quality of service (QoS) provision for Staffordshire University’s network communications infrastructure (SUNCI) providing services to over 10000 users. The authors discuss a new approach to evaluate the SUNCI QoS provision, by adding a component of measurements to the existing model presented by Walker [1]. The authors utilize the current SUNCI, as a testing platform to assess QoS provision in the light of users’ perception, and compare it with the network traffic measurements and online monitoring reports. Users’ perception of telecommunications’ network infrastructure QoS provision is critical to successful business management operation of any organization. This paper offers a cost effective approach to assess the QoS provision within the Staffordshire University (SU) campus network, which could be easily adapted to any university campus network in the world. The paper show the evidence that there is a good correlation between the users’ perception acquired via a survey questionnaire and the network measurements results.

Performance benchmarking of wireless Web servers

The advent of mobile computers and wireless networks enables the deployment of wireless Web servers and clients in short-lived ad hoc network environments, such as classroom area networks. The purpose of this paper is to benchmark the performance capabilities of wireless Web servers in such an environment. Network traffic measurements are conducted on an in-building IEEE 802.11b wireless ad hoc network, using a wireless-enabled Apache Web server, several wireless clients, and a wireless network traffic analyzer. The experiments focus on the HTTP transaction rate and end-to-end throughput achievable in such an ad hoc network environment, and the impacts of factors such as Web object size, number of clients, and persistent HTTP connections. The results show that the wireless network bottleneck manifests itself in several ways: inefficient HTTP performance, client-side packet losses, server-side packet losses, network thrashing, and unfairness among Web clients. Persistent HTTP connections offer up to 350% improvement in HTTP transaction rate and user-level throughput, while also improving fairness for mobile clients accessing content from a wireless Web server.