Internet Traffic Data Research Articles

Anomaly Detection and Attribution in Networks With Temporally Correlated Traffic

Anomaly detection in communication networks is the first step in the challenging task of securing a network, as anomalies may indicate suspicious behaviors, attacks, network malfunctions, or failures. In this paper, we address the problem of not only detecting the anomalous events but also of attributing the anomaly to the flows causing it. To this end, we develop a new statistical decision theoretic framework for temporally correlated traffic in networks via Markov chain modeling. We first formulate the optimal anomaly detection problem via the generalized likelihood ratio test (GLRT) for our composite model. This results in a combinatorial optimization problem which is prohibitively expensive. We then develop two low-complexity anomaly detection algorithms. The first is based on the cross entropy (CE) method, which detects anomalies as well as attributes anomalies to flows. The second algorithm performs anomaly detection via GLRT on the aggregated flows transformation - a compact low-dimensional representation of the raw traffic flows. The two algorithms complement each other and allow the network operator to first activate the flow aggregation algorithm in order to quickly detect anomalies in the system. Once an anomaly has been detected, the operator can further investigate which specific flows are anomalous by running the CE-based algorithm. We perform extensive performance evaluations and experiment our algorithms on synthetic and semi-synthetic data, as well as on real Internet traffic data obtained from the MAWI archive, and finally make recommendations regarding their usability.

Positive and Unlabeled Learning for User Behavior Analysis Based on Mobile Internet Traffic Data

With the rapid development of wireless communication and mobile Internet, mobile phone becomes ubiquitous and functions as a versatile and a smart system, on which people frequently interact with various mobile applications (Apps). Understanding human behaviors using mobile phone is significant for mobile system developers, for human-centered system optimization and better service provisioning. In this paper, we focus on mobile user behavior analysis and prediction based on mobile Internet traffic data. Real traffic flow data is collected from the public network of Internet service providers, by high-performance network traffic monitors. We construct a User-App bipartite network to represent the traffic interaction pattern between users and App servers. After mining the explicit and implicit features from the User-App bipartite network, we propose two positive and unlabeled (PU) learning methods, including Spy-based PU learning and K-means-based PU learning, for App usage prediction and mobile video traffic identification. We first use the traffic flow data of QQ, a very famous messaging and social media application possessing high market share in China, as the experimental data set for App usage prediction task. Then, we use the traffic flow data from six popular Apps, including video intensive Apps (Youku, Baofeng, LeTV, and Tudou) and other Apps (Meituan and Apple), as the experimental data set for mobile video traffic identification task. Experimental results show that our proposed PU learning methods perform well in both tasks.

A new probabilistic classifier based on decomposable models with application to internet traffic

Probabilistic models are one of the common approaches for classification. One way to create these models is to form a decomposable model by selecting a set of marginal distributions. Although decomposable models are attractive by having some desirable properties, they would face the over-fitting issue in model-based classification approaches. Considering this issue, we propose a new method for selecting a set of marginal distributions and creating a proper decomposable model while controlling the complexity. The obtained model will be able to capture the interdependencies among different features and can be used for classification. The proposed method is compared with three existing methods namely TAN and Averaged TAN classifiers and t-Cherry algorithm by focusing on internet traffic data. Experimental results show that our obtained model can effectively extract dependencies among features, and hence, its performance as a classifier is superior compared to other three methods.

Fast Tensor Factorization for Accurate Internet Anomaly Detection

Detecting anomalous traffic is a critical task for advanced Internet management. Many anomaly detection algorithms have been proposed recently. However, constrained by their matrix-based traffic data model, existing algorithms often suffer from low accuracy in anomaly detection. To fully utilize the multi-dimensional information hidden in the traffic data, this paper takes the initiative to investigate the potential and methodologies of performing tensor factorization for more accurate Internet anomaly detection. More specifically, we model the traffic data as a three-way tensor and formulate the anomaly detection problem as a robust tensor recovery problem with the constraints on the rank of the tensor and the cardinality of the anomaly set. These constraints, however, make the problem extremely hard to solve. Rather than resorting to the convex relaxation at the cost of low detection performance, we propose TensorDet to solve the problem directly and efficiently. To improve the anomaly detection accuracy and tensor factorization speed, TensorDet exploits the factorization structure with two novel techniques, sequential tensor truncation and two-phase anomaly detection. We have conducted extensive experiments using Internet traffic trace data Abilene and GEANT. Compared with the state of art algorithms for tensor recovery and matrix-based anomaly detection, TensorDet can achieve significantly lower false positive rate and higher true positive rate. Particularly, benefiting from our well designed algorithm to reduce the computation cost of tensor factorization, the tensor factorization process in TensorDet is 5 (Abilene) and 13 (GEANT) times faster than that of the traditional Tucker decomposition solution.

CHARACTERIZATION OF UNIVARIATE LONG-TERM URBAN INTERNET TRAFFIC VOLUME

The proposed work deals with a real time hourly internet traffic data set in bits collected from ISPs located in 11 cities of European Country for the period 7 th June 2005 to 31 st July 2005. Then a thorough statistical inference has been drawn regarding the central tendency, dispersion and distribution of the data. Time-frequency analysis using Smoothed Pseudo Wigner Ville Distribution (SPWVD) is implied to infer knowledge about the non-stationarity of the system. A nonparametric test for normality, Anderson Darling Test (AD-Test) has been performed to detect the binary signature of nonlinearity in the signal. Delay Vector Variance Analysis (DVV) are being exploited to infer deeper knowledge about the determinism and nonlinearity in the system. The results confirm a nonstationary, relatively stochastic and nonlinear profile of the signal under observation.

Enhanced web cache replacement policy based on data mining and RFSD scoring

By rapid growth of the Internet users and devices, the number of servers also increase simultaneously which causes the exponential increment of the internet traffic and static data. Handling these huge amounts of user requests and efficiently responding to them require high bandwidth links, powerful servers and robust equipment, which despite the availability of these requirements getting the full user satisfaction is extremely difficult and a tough challenge. In order to overcome the mentioned problem, the cache servers are being used as a suitable solution. The performance of web cache server directly depends on its replacement policies. Several cache replacement policies have been proposed in literature each having varied hit rate (HR) and byte hit rate (BHR) performances on different networks. The replacement policy proposed in this paper is a dynamic cache replacement policy which trains itself utilizing previous network logs and by exploiting the data mining clustering algorithm. Once the training step is completed, the proposed policy utilizes the normalization formulas to score each metric of the enquiries including recency, frequency, size and delay. Simulation results showed that the proposed policy has the optimum performance on different networks and it not only improved the performance of web cache server in term of HR and BHR, but also decreases the data retrieval time (Delay Ratio (DR)) of the cache servers.

Efficient and robust feature extraction and selection for traffic classification

Given the limitations of traditional classification methods based on port number and payload inspection, a large number of studies have focused on developing classification approaches that use Transport Layer Statistics (TLS) features and Machine Learning (ML) techniques. However, classifying Internet traffic data using these approaches is still a difficult task because (1) TLS features are not very robust for traffic classification because they cannot capture the complex non-linear characteristics of Internet traffic, and (2) the existing Feature Selection (FS) techniques cannot reliably provide optimal and stable features for ML algorithms. With the aim of addressing these problems, this paper presents a novel feature extraction and selection approach. First, multifractal features are extracted from traffic flows using a Wavelet Leaders Multifractal Formalism(WLMF) to depict the traffic flows; next, a Principal Component Analysis (PCA)-based FS method is applied on these multifractal features to remove the irrelevant and redundant features. Based on real traffic traces, the experimental results demonstrate significant improvement in accuracy of Support Vector Machines (SVMs) comparing to the TLS features studied in existing ML-based approaches. Furthermore, the proposed approach is suitable for real time traffic classification because of the ability of classifying traffic at the early stage of traffic transmission.

Analisis Pemanfaatan Internet Di Pemerintah Kota Denpasar

The usage of the Internet can be described by the traffic characteristics, user behavior, and bandwidth. This research collected Internet traffic data at the government office of Denpasar city in October 2015, for study case of Dinas Perijinan. The result shows the average 90 % volume of TCP protocol and 10 % of others, such as UDP, GRE, and ICMP. There are average 95 % of application port’s volumes were used by IANA’s port, and 5 % by unsigned port, which means it may become a potential to be used as malware and file sharing application. The biggest domain’s volumes were accessed to Youtube and Facebook websites, and also to other malware domain. Based on the observation of user's behaviour, there are 3 of 12 sample users did not understand how to access good link from malware. From the analysis, a limitation access on youtube is one of recommendation actions besides closing malware port and user training.

Dynamic Router Performance Control Utilizing Support Vector Machines for Energy Consumption Reduction

In this paper, we propose a machine-learning-based novel dynamic performance control method for routers that supports several performance levels. The method utilizes support vector machine (SVM) to determine performance level change points. We utilize a traffic normalization technique with a corresponding performance threshold that allows us to apply the same SVM to different traffic volumes. This technique shortens the learning process for control. Several experiments on real Internet traffic data sequences prove that the method yields higher energy efficiency than conventional methods, that is, our previously proposed frequency decomposition-based method and a conventional traffic prediction method based on auto-regressive moving average model with optimal parameter values given by Akaike’s information criterion. We also evaluate the impact of several key parameters such as traffic measurement interval and the number of packet processing engines. The results clarify the proper ranges of parameter values that attain significant power reductions while keeping packet loss to acceptable levels.

Modeling Internet Traffic Generations Based on Individual Users and Activities for Telecommunication Applications

A traffic generation model is a stochastic model of the data flow in a communication network. These models are useful during the development of telecommunication technologies and for analyzing the performance and capacity of various protocols,algorithms, and network topologies. We present here two modeling approaches for simulating internet traffic. In our models, we simulate the length and interarrival times of individual packets, the discrete unit of data transfer over the internet. Our first modeling approach is based on fitting data to known theoretical distributions. The second method utilizes empirical copulae and is completely data driven. Our models were based on internet traffic data generated by different individuals performing specific tasks (e.g., web browsing, video streaming, and online gaming). When combined, these models can be used to simulate internet traffic from multiple individuals performing typical tasks. KEYWORDS: Internet Traffic Simulation; Stochastic Models; Empirical Copula; Cumulative distribution function; Wireshark

Imbalanced traffic identification using an imbalanced data gravitation-based classification model

As a basic method for monitoring the activities of Internet applications, traffic identification is very important for Internet management and security. Internet traffic data naturally exhibits imbalanced distributions, but this problem is rarely considered by the research community. Data gravitation-based classification (DGC) is a new classification model for handling imbalance data sets and we proposed an imbalanced DGC (IDGC) model in our previous study. In the present study, we developed an IDGC-based model to resolve imbalanced Internet traffic identification problems. First, we constructed six imbalanced traffic data sets from three original traffic data sets and we then extracted their early stage features according to the packet sizes. In identification experiments, we compared the performance of six standard algorithms, including DGC and four imbalanced algorithms with IDGC. The experimental results demonstrated that the standard classification models could achieve high accuracy with imbalanced traffic data, but their imbalanced performance was not as good, and their generalizability was also a problem. By contrast, IDGC performed very well and in a stable manner in terms of imbalanced performance measures, thereby demonstrating its effectiveness for imbalanced traffic identification.

How Essential and Biased are Search Engines? Empirical Evidence from the Flow of Internet Traffic

Dominant horizontal search engines have recently been facing antitrust challenges around the globe based on allegations that their search results are biased. In this short article, we present empirical evidence based on Internet traffic data from Korea that weakly supports this search bias claim. In short, we show that search engines tend to herd their users back to their own content significantly more than their share of overall page views. Moreover, about a quarter of all page views originating from outside sources come from search engines, thus suggesting that search engines are fairly important to most websites. However, results of this study do not fully justify immediate antitrust intrusion. Instead, they may be used as an initial test for a more thorough scrutiny.

Has the Dominant Search Engine, If Any, Discriminated Against Rival Websites?

Some dominant horizontal search engines have recently been facing antitrust charges worldwide, based on allegations that they give preference to their own contents and discriminate against their rivals. In this short article, we use Internet traffic data to test whether the entry of a dominant search engine to certain downstream Internet service markets has affected the outflow of Internet traffic from this search engine to its downstream rivals. We do not find any empirical evidence of such a search bias.

Performance Comparison of Four New ARIMA-ANN Prediction Models on Internet Traffic Data

Prediction of Internet traffic time series data (TSD) is a challenging research problem, owing to the complicated nature of TSD. In literature, many hybrids of auto-regressive integrated moving average (ARIMA) and artificial neural networks (ANN) models are devised for the TSD prediction. These hybrid models consider such TSD as a combination of linear and non-linear components, apply combination of ARIMA and ANN in some manner, to obtain the predictions. Out of the many available hybrid ARIMA-ANN models, this paper investigates as to which of them suits better for Internet traffic data. This suitability of hybrid ARIMAANN models is studied for both one-step ahead and multistep ahead prediction cases. For the purpose of the study, Internet traffic data is sampled at every 30 and 60 minutes. Model performances are evaluated using the mean absolute error and mean square error measurement. For one-step ahead prediction, with a forecast horizon of 10 points and for three-step prediction, with a forecast horizon of 12 points, the moving average filter based hybrid ARIMA-ANN model gave better forecast accuracy than the other compared models.

Effective packet number for early stage internet traffic identification

Accurately identifying Internet traffic at the early stage is very important for the applications of traffic identification. Recent years, more and more research works have tried to build effective machine learning models to identify an Internet flow with the few packets at its early stage. However, a basic and important problem still needs to be studied in depth, that is how many packets are most effective in early stage Internet traffic identification. In this paper, we try to resolve this problem. Three Internet traffic data sets are applied. And the sizes of the first 10 packets are extracted for study. We firstly apply mutual information to analyze the information that the first n packets provide to the flow type. Then correlation analysis of each pair of adjacent packets is carried out to find out the feature redundancies. And then we execute a number of crossover identification experiments with different numbers of packets using 11 well-known supervised learning algorithms. Finally, statistical tests are applied for the experimental results to find out which number is the best performed one. Our experimental results show that 5–7 are the best packet numbers for early stage traffic identification.

Identifying elephant flows in internet backbone traffic with bloom filters and LRU

Traffic measurements provide critical input for network security, traffic engineering and accounting. Restrained capabilities of computing and storage have motivated recent researches on partial flow maintenance like identifying elephant flows. Considering high false negative probability of traditional algorithms, a novel scheme called BF–LRU (Bloom filters and least recent used) is presented. Our BF–LRU scheme adopts LRU replacement to evict mice flows and Bloom filters representation to conserve heavy hitters. Based on Pareto and hypergeometric distribution, expressions of upper-bound error probability are analyzed in detail. Experiments are conducted based on real Internet traffic data. Simulation results indicate that BF–LRU can not only achieve lower error probability, but also scale up to OC-768 backbone trace without losing any space efficiency.

Designing an Internet Traffic Predictive Model by Applying a Signal Processing Method

Detection of abnormal internet traffic has become a significant area of research in network security. Due to its importance, many predictive models are designed by utilizing machine learning algorithms. The models are well designed to show high performances in detecting abnormal internet traffic behaviors. However, they may not guarantee reliable detection performances for new incoming abnormal internet traffic because they are designed using raw features from imbalanced internet traffic data. Since internet traffic is non-stationary time-series data, it is difficult to identify abnormal internet traffic with the raw features. In this study, we propose a new approach to detecting abnormal internet traffic. Our approach begins with extracting hidden, but important, features by utilizing discrete wavelet transformation. Then, statistical analysis is performed to filter out irrelevant and less important features. Only statistically significant features are used to design a reliable predictive model with logistic regression. A comparative analysis is conducted to determine the importance of our approach by measuring accuracy, sensitivity, and the Area Under the receiver operating characteristic Curve. From the analysis, we found that our model detects abnormal internet traffic successfully with high accuracy.

Current Trends and Controversies in Internet Peering and Transit: Implications for the Future Evolution of the Internet

The hierarchy of Internet peering and transit interconnection agreements between Internet Service Providers (ISPs) defines the core economic structure of the Internet. Recent interconnection agreements (e.g., between Netflix and AT&T, Comcast or Verizon) have generated considerable comment and concern, much of which is misleading as to the root causes for new peering agreements and even the future of the Internet itself. Given this critical setting, this paper examines how and why Internet peering and transit is changing, and discusses the implications of these changes on the future evolution of the Internet. What makes this research novel is the analysis of new Internet traffic data to identify future trends relevant to Internet interconnection. These data provide insight into the congestion level of different interconnects in a few places on the Internet. We next develop a framework to describe past evolution in Internet Interconnection in order to better understand the changing characteristics in Internet traffic, and other new developments impacting the economics of Internet interconnection. This framework demonstrates how most network providers have been progressively expanding their networks over a larger proportion of the end-to-end connection (i.e., increasing their network investment), providing an explanation of the changes in Internet interconnection, and resulting controversies, to keep pace with this network evolution. The strategic implications of our analysis: 1) a more complete understanding of the full network investment undertaken by network providers may help avoid, or at least better explain, the future controversies, 2) rapidly changing nature of Internet content will be constant source of change in interconnection, and 3) the changing role of Tier 1 providers will be further impacted by increasing peering among Tier 2 and Tier 3 providers through the implementation of remote peering based upon open standards for Internet Exchange Providers. Taken as a whole, the increasingly overlapping roles of network providers call into question the utility of the traditional Tiered framework as a useful description of the current Internet peering ecosystem, and suggests the need for a new, updated framework to predict and describe future peering relationships.

Structural insights on Internet traffic: Community overlapping and correlations

There is an increasing number of Internet applications, which leads to an increasing network capacity and availability. Internet traffic characterisation and application identification are, therefore, more important for efficient network management. In this paper, we construct flow graphs from detailed Internet traffic data collected from the public networks of Internet Service Providers. We analyse the community structures of the flow graph that is naturally formed by different applications. The community size, degree distribution of the community, and community overlap of 10 Internet applications are investigated. We further study the correlations between the communities from different applications. Our results provide deep insights into the behaviour Internet applications and traffic, which is helpful for both network management and user behaviour analysis.

Special Section Introduction: Cloud Computing as a Service

Cloud computing is a new way of delivering computing services. The flexibility, scalability, and cost effectiveness of cloud computing provides tremendous business opportunities to customers. The elated customer expectation, however, poses daunting challenges to cloud service providers, whether they are traditional data centers or fresh players in the arena. These practical challenges raise new research questions to communities, both computer science oriented and service science oriented. Performance analysis and service management are required to be knitted together for the success of the end game. This special section aims to facilitate the development of new theories as well as their permeation to the practice fields. We provide a platform for both researchers and practitioners to communicate their new ideas so that important problems in cloud computing services can be addressed from a more complete point of view. Various aspects of cloud computing service have been discussed in the set of articles presented here. In particular, we are pleased that several studies involve the development of new methodology and concepts in fields such as statistics, computer science, operations research, and marketing science. The Iyoob et al. (2013) paper seeks to understand the role that operations research (OR) plays in cloud computing. The authors effectively organize their research questions into three areas according to host: (1) cloud providers, (2) cloud consumers, and (3) cloud brokers. They conducted a thorough survey on existing research in this area. The authors argue that cloud computing, as a new service form, could be a fresh force that stimulates new research in OR. More specifically, the authors identify several less developed and more challenging areas, such as pricing (from the provider’s perspective), intrusion detection and prevention (from the provider’s perspective), and capacity reverse auctions (from the broker’s perspective). The authors expect new and exciting research in these areas. In the paper by Bohling et al. (2013), the problem of cloud computing service adoption is viewed in a broad perspective of radical innovation adoption, an innovative addition to the fundamental research on relationship marketing. More specifically, several important decisions in adopting cloud computing services, such as purchase timing, product choice, and purchase amount, have been analyzed, and a new construct for explaining customer behavior has been added to the theory of the relationship marketing. Numerical experiments provide validation of the model, as well as interesting observations. The paper by Lee et al. (2013) showcases a recent application that integrates advanced energy analytics and cloud computing service. Sophisticated analytic tools have been developed for analyzing and controlling the building energy consumption. Energy consumption reduction can be maximized only when all the parties involved have access to the information and each contributes his or her part in the equation. The cloud service provides a venue for the parties to access the desired information. It enables building owners, facility managers, operators, and tenants to access necessary analytic tools with no need to install the tools themselves. In this application, cloud service is indeed a key enabler of the technology. The paper by Gaivoronski et al. (2013) introduces several portfolio optimization models from the perspective of cloud brokers. This study fills the gap between the technical aspect and the economic/financial aspect of cloud brokering. A set of models is developed to find the best portfolio of connectivity options that maximizes the expected profit of the cloud broker while satisfying a constraint on the quality of service (QoS). The novelty of these models lies in the fact that they not only incorporate fixed costs but also consider the uncertainty in demand and QoS. Theoretical analysis and business insights are delivered through numerical experiments, utilizing observations from real Internet traffic data.