Traffic Matrix Estimation Problem Research Articles

Structured nonnegative matrix factorization for traffic flow estimation of large cloud networks

Network traffic matrix estimation is an ill-posed linear inverse problem: it requires to estimate the unobservable origin destination traffic flows, X, given the observable link traffic flows, Y, and a binary routing matrix, A, which are such that Y=AX. This is a challenging but vital problem as accurate estimation of OD flows is required for several network management tasks. In this paper, we propose a novel model for the network traffic matrix estimation problem which maps high-dimension OD flows to low-dimension latent flows with the following three constraints: (1) nonnegativity constraint on the estimated OD flows, (2) autoregression constraint that enables the proposed model to effectively capture temporal patterns of the OD flows, and (3) orthogonality constraint that ensures the mapping between low-dimensional latent flows and the corresponding link flows to be distance preserving. The parameters of the proposed model are estimated with a training algorithm based on Nesterov accelerated gradient and generally shows fast convergence. We validate the proposed traffic flow estimation model on two real backbone IP network datasets, namely Internet2 and GÉANT. Empirical results show that the proposed model outperforms the state-of-the-art models not only in terms of tracking the individual OD flows but also in terms of standard performance metrics. The proposed model is also found to be highly scalable compared to the existing state-of-the-art approaches.

Learning Based Methods for Traffic Matrix Estimation From Link Measurements

Network traffic matrix (TM) is a critical input for capacity planning, anomaly detection and many other network management related tasks. The TMs are often computed from link load measurements. The TM estimation problem is the determination of the TM from link load measurements. The relationship between the link loads and the TM that generated the link loads can be modeled as an under-determined linear system and has multiple feasible solutions. Therefore, prior knowledge of the traffic demand pattern has to be used in order to find a potentially feasible TM. In this paper, we consider the TM estimation problem with limited prior information. Unlike previous methods that require past measurements of complete TMs, which are hard to obtain or protected by regulations, our method works even if only the distribution of TMs is known. We develop an iterative projection based algorithm to solve this problem. If large number of past TMs can be measured, we propose a Generative Adversarial Network (GAN) based approach for solving the problem. We compare the strengths of the two approaches and evaluate their performance for several networks using varying amounts of past data.

An SDN-Based Traffic Matrix Estimation Framework

Accurate traffic matrix estimation is critical to solving many networking problems, such as routing and network provisioning, etc. This task in a traditional IP network is tackled with link load measurements. But the accuracy of such an approach is low because the underlying system of linear equations governing the traffic estimation problem is highly under-determined in this approach. A software-defined network (SDN) provides measurements of more types of flows, and thus opens up new opportunities for tackling the traffic matrix estimation problem. Furthermore, the controller of an SDN can dynamically install flows in the flow table of an SDN router for traffic measurement. But there is one constraint: the number of entries in a flow table for traffic measurement is limited because the flow table in an SDN node is usually constructed with expensive ternary content addressable memory (TCAM) devices. How to use the limited number of entries in a flow table for traffic measurement is a challenging task in SDNs. In this paper, we present a new framework for solving the traffic matrix estimation problem in an SDN-based IP network. An important feature of the proposed framework is that any flow added to the flow table of an SDN router can increase the rank of the underlying equations governing the traffic matrix estimation problem. This greatly improves the utilization efficiency of TCAM entries for traffic measurement. Detailed performance evaluation given in this paper demonstrates that the proposed approach can achieve significant performance gains over other approaches.

A PCA based optimization approach for IP traffic matrix estimation

Inferring traffic matrix (TM) from link measurements and routing information has important applications including capacity planning, traffic engineering and network reliability analysis. The challenge comes from that there are more unknowns than data. To face this challenge, this paper describes the inference problem as an optimization problem, where the objective is to minimize the Mahalanobis distance between the solution and a certain prior distribution, subject to the routing and link measurement constraints. This optimization problem is then solved by the Moore–Penrose inverse of the routing matrix. To reduce the computing complexity, a principal component analysis (PCA) approach is further applied in solving the optimization problem. We obtain the explicit formulas by using the Moore–Penrose inverse and the PCA theory. On the basis of the generalized inverse of routing matrix and the PCA theory, we propose an interesting generalized Tomogravity approach, which is subsequently termed as PCAOM. We present the complete mathematical solution and the algorithm of the described TM estimation problem. By introducing a weight parameter, a generalized algorithm is presented, which can be applied flexibly by adjusting the importance of the prior according to the accuracy of the prior or even no prior is required when the prior is unavailable. Numerical results are provided to demonstrate the accuracy of our method with the dataset of Abilene network through the comparison with the famous Tomogravity method. Given that we have proposed two algorithms for the optimization problem of TM estimation, we also provide a guideline on how to choose the proper algorithm according to the availability of the prior information.

A convex optimization-based traffic matrix estimation approach in IP-over-WDM backbone networks

With the rapid development of an IP-over-WDM network, it has become a complex and heterogeneous network. In such a case, network management is a crucial role for promoting the efficiency of the IP-over-WDM network. A traffic matrix, which provides an intrinsic and flow-level view of our networks, is required to perform network management tasks such as traffic engineering function. Unfortunately, limited by current technologies, it is significantly difficult to acquire the exact traffic matrix. Motivated by this issue, in this paper, we investigate the traffic matrix estimation problem in an IP-over-WDM backbone network. We develop a simple algorithm based on the network tomography method. In order to deal with the ill-posed property of the traditional network tomography method, we refer to the origin–destination traffic demands of the optical layer in our method. We first study the relationship between the traffic matrix and optical layer traffic demands. Thereby, we take advantage of the relationship between them to construct a linear system. Combining with the traditional network tomography model, we propose a convex and unconstrained optimization model for estimating the traffic matrix. Finally, we also provide numerical results to validate the performance of the proposed method in this literature.

A compressive sensing‐based approach to end‐to‐end network traffic reconstruction utilising partial measured origin‐destination flows

AbstractTraffic matrices describe the volume of traffic flowing from origin nodes to destination nodes. As one of the most crucial input parameters for network activities, it is employed to perform a great number of network management tasks. However, traffic matrix estimation is unavailable and has a highly ill‐posed nature, which means that it is significantly difficult to ensure the accuracy for estimating traffic matrices. Thus, it is a challenge to obtain the precise end‐to‐end network traffic. In this paper, we investigate the problem of network traffic estimation and propose a novel compressive sensing‐based approach using partial measured origin‐destination (OD) flows. As a purely data‐driven method, we first propose a strategy to select a small subset of the measured OD flows. We then put forward an optimal greedy adaptive dictionary learning algorithm in order to make the traffic matrix sparse. Furthermore, we model the traffic matrix estimation problem by a fundamental inference problem, which is ill‐posed and obeys the constraints of compressive sensing. In other words, the estimation model is a convex optimisation problem. In the process, we validate the performance of our method by real traffic data. Simulation results indicate that our method can trace the shape of each OD flow much more precisely. Copyright © 2014 John Wiley & Sons, Ltd.

Coarse-grained traffic matrix estimation for data center networks

We address the problem of estimating the real-time traffic flows in data center networks (DCNs), using the light-weight SNMP data. Unlike the problem of estimating the traffic matrix (TM) across origin–destination (OD) pairs in ISP networks, the traffic flows across servers or Top of Rack (ToR) switch pairs in DCNs are notoriously more irregular and volatile. Although numerous methods have been proposed in past several years to solve the TM estimation problem in ISP networks, none of them could be applied to DCNs directly. In this paper, we make the first step to solve the TM estimation problem in DCNs by leveraging the characteristics of prevailing data center architectures and decomposing the topologies of DCNs, which makes TM estimation problems in DCNs easy to handle. We also state a basic theory to obtain the aggregate traffic characteristics of these clusters unbiasedly. We propose two efficient TM estimation algorithms based on the decomposed topology and the aggregate traffic information, which improves the state-of-the-art tomography methods without requiring any additional instrumentation. Finally, we compare our proposal with a recent representative TM estimation algorithm through both real experiments and extensive simulations, the results show that, (i) the data center TM estimation problem could be well handled after the decomposition step, (ii) our two algorithms outperforms the former one in both speed and accuracy.

Joint time–frequency sparse estimation of large-scale network traffic

When 3G, WiFi, and WiMax technologies are successfully applied to access networks, current communication networks become more and more complex, more and more heterogeneous, and more difficult to manage. Moreover, network traffic exhibits the increasing diversities and concurrently shows many new characteristics. The real-time end-to-end demand urges network operators to learn and grasp traffic matrix covering their networks. However, unfortunately traffic matrix is significantly difficult directly to attain. Despite many studies made previously about traffic matrix estimation problem, it is a significant challenging to obtain its reliable and accurate solution. Here we propose a novel approach to solve this problem, based on joint time–frequency analysis in transform domain. Different from previous methods, we analyze the time–frequency characteristics about traffic matrix and build the time–frequency model describing it. Generally, traffic matrix can be divided into tendency terms and fluctuation terms. We find that traffic matrix in time–frequency domain owns the more obvious sparsity than in time domain. Obviously, its tendency terms and fluctuation terms also have the lower dimensions in time–frequency domain. This brings us into the field of compressive sensing that is a generic technique for data reconstruction. Additionally, we take into account updating time–frequency model presented with link loads to make our model adaptive. Finally, comparative analysis in two real backbone networks confirms that the accuracy, stability, and effectiveness of our approach.