Operation Of Large Networks Research Articles

Convex Heuristics for Optimal Placement and Operation of Valves and Chlorine Boosters in Water Networks

This paper investigates the problem of optimal placement and operation of valves and chlorine boosters in water networks. The objective is to minimize average zone pressure while penalizing deviations from target chlorine concentrations. The problem formulation includes nonconvex quadratic terms within constraints representing the energy conservation law for each pipe, and discretized differential equations modeling advective transport of chlorine concentrations. Moreover, binary variables model the placement of valves and chlorine boosters. The resulting optimization problem is a nonconvex mixed integer nonlinear program, which is difficult to solve, especially when large water networks are considered. We develop a new convex heuristic to optimally place and operate valves and chlorine boosters in water networks, while estimating the optimality gaps for the computed solutions. We evaluate the proposed heuristic using case studies with varying sizes and levels of connectivity and complexity, including two large operational water networks. The convex heuristic is shown to generate good-quality feasible solutions in all problem instances with bounds on the optimality gap comparable to the level of uncertainty inherent in hydraulic and water quality models.

Optimizing in the Dark: Learning Optimal Network Resource Reservation Through a Simple Request Interface

Network resource reservation systems are being developed and deployed, driven by the demand and substantial benefits of providing performance predictability for modern distributed applications. However, existing systems suffer limitations: They either are inefficient in finding the optimal resource reservation, or cause private information (e.g., from the network infrastructure) to be exposed (e.g., to the user). In this paper, we design BoxOpt, a novel system that leverages efficient oracle construction techniques in optimization and learning theory to automatically, and swiftly learn the optimal resource reservations without exchanging any private information between the network and the user. In BoxOpt, we first model the simple reservation interface adopted in most reservation systems as a resource membership oracle. Second, we develop an efficient algorithm that constructs a resource separation oracle by a linear number of calls on resource membership oracle. Third, we develop a generic framework to construct a resource optimization oracle by iteratively calling the resource separation oracle, and then develop three novel, efficient algorithms under this generic framework, the best of which computes the optimal resource reservation by a linear number of calls on resource separation oracle. As such, BoxOpt can discover the optimal resource reservation with O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) calls on the resource membership oracle. We implement a prototype of BoxOpt with and demonstrate its efficiency and efficacy via extensive experiments using real network topology and a 7-day trace from a large operational federation network. Results show that (1) BoxOpt has a 100% correctness ratio by comparing with a state-of-the-art optimization solver, and (2) for 90% of requests, BoxOpt learns the optimal resource reservation within 10 seconds.

Optimal control of water distribution networks without storage

The paper investigates the problem of optimal control of water distribution networks without storage capacity. Using mathematical optimization, we formulate and solve the problem as a non-convex NLP, in order to obtain optimal control curves for both variable speed pumps and pressure reducing valves of the network and thus propose a methodology for the automated control of real operational networks. We consider both single-objective and multi-objective problems with average zonal pressure, pump energy consumption and water treatment cost as objectives. Furthermore, we investigate global optimality bounds for the calculated solutions using global optimization techniques. The proposed approach is shown to outperform state-of-the-art global optimization solvers. The described procedure is demonstrated in a case study using a large size operational network.

Hydraulically informed graph theoretic measure of link criticality for the resilience analysis of water distribution networks

Water Distribution Networks (WDN) are complex and highly interconnected systems. To maintain operation under failure conditions, WDNs should have built-in resilience based on topological and energy redundancy. There are various methods for analysing the resilience of WDNs based on either hydraulic models or surrogate network measures; however, not a single universally accepted method exists. Hydraulic modeling of disruptive operational scenarios suffer from combinatorial restrictions and uncertainties. Methods that rely on surrogate network measures do not take into account the complex interactions between topological and energy redundancy. To bridge this gap, the presented work introduces a hydraulically informed surrogate measure of pipe criticality for the resilience analysis of WDNs, called Water Flow Edge Betweenness Centrality (WFEBC). The WFEBC combines the random walk betweenness centrality with hydraulic (energy) loss principles in pipes. The proposed network resilience estimation method is applied to a case study network and an operational network. Furthermore, a network decomposition approach is proposed to complement the network estimation method and facilitate its scalability to large operational networks. The described resilience analysis method is benchmarked against a hydraulic model-based analysis of WDN reserve capacity. WFEBC is also applied to assess the improvement in resilience allowed by the implementation of a dynamically adaptive topology in an operational network. The benefits and limitations of the proposed method are discussed.

Predictive Impact Analysis for Designing a Resilient Cellular Backhaul Network

Backhaul transport network design and optimization for cellular service providers involve a unique challenge stemming from the fact that an end-user's equipment (UE) is within the radio reach of multiple cellular towers: It is hard to evaluate the impact of the failure of the UE's primary serving tower on the UE, because the UE may simply switch to get service from other nearby cellular towers. To overcome this challenge, one needs to quantify the cellular service redundancy among the cellular towers riding on that transport circuit and their nearby cellular towers, which in turn requires a comprehensive understanding of the radio signal profile in the area of the impacted towers, the spatial distribution of UEs therein, and their expected workload (e.g., calls, data throughput). In this work, we develop a novel methodology for assessing the service impact of any hypothetical cellular tower outage scenario, and implement it in an operational system named Tower Outage Impact Predictor (TOIP). Our evaluations, using both synthetic data and historical real tower outages in a large operational cellular network, show conclusively that TOIP gives an accurate assessment of various tower outage scenarios, and can provide critical input data towards designing a reliable cellular backhaul transport network.

Predictive Impact Analysis for Designing a Resilient Cellular Backhaul Network

Backhaul transport network design and optimization for cellular service providers involve a unique challenge stemming from the fact that an end-user's equipment (UE) is within the radio reach of multiple cellular towers: It is hard to evaluate the impact of the failure of the UE's primary serving tower on the UE, because the UE may simply switch to get service from other nearby cellular towers. To overcome this challenge, one needs to quantify the cellular service redundancy among the cellular towers riding on that transport circuit and their nearby cellular towers, which in turn requires a comprehensive understanding of the radio signal profile in the area of the impacted towers, the spatial distribution of UEs therein, and their expected workload (e.g., calls, data throughput). In this work, we develop a novel methodology for assessing the service impact of any hypothetical cellular tower outage scenario, and implement it in an operational system named Tower Outage Impact Predictor (TOIP). Our evaluations, using both synthetic data and historical real tower outages in a large operational cellular network, show conclusively that TOIP gives an accurate assessment of various tower outage scenarios, and can provide critical input data towards designing a reliable cellular backhaul transport network.

Inter-Controller Traffic to Support Consistency in ONOS Clusters

In distributed SDN architectures, the network is controlled by a cluster of multiple controllers. This distributed approach permits to meet the scalability and reliability requirements of large operational networks. Despite that, a logical centralized view of the network state should be guaranteed, enabling the simple development of network applications. Achieving a consistent network state requires a consensus protocol, which generates control traffic among the controllers whose timely delivery is crucial for network performance. We focus on the state-of-art ONOS controller, designed to scale to large networks, based on a cluster of self-coordinating controllers. In particular, we study the inter-controller control traffic due to the adopted consistency protocols. Based on real traffic measurements and the analysis of the adopted consistency protocols, we develop some empirical models to quantify the traffic exchanged among the controllers, depending on the considered shared data structures, the current network state (e.g., topology) and the occurring network events (e.g., flow or host addition). Our models provide a formal tool to be integrated into the design and dimension the control network interconnecting the controllers. Our results are of paramount importance for the proper design of large SDN networks, in which the control plane is implemented in-band and cannot exploit dedicated network resources.

Optimized Control of Pressure Reducing Valves in Water Distribution Networks with Dynamic Topology

A dynamic topology aggregates zones in water distribution networks (WDNs) for improved pressure management and resilience to failure. Based on a sequential convex programming (SCP) approach, we propose an optimization method for the control of pressure reducing valves (PRV) in WDNs with dynamic topology. By restricting the SCP iterations to the feasible search space, we show that reliable convergence of the method is achieved. Using an experimental study in a large operational network, the optimization of PRV settings with a dynamic topology is shown to result in pressure reductions of 3.7% compared to optimized PRVs in a closed DMA structure.

A quality metric for IDS signatures: in the wild the size matters

The manual forensics investigation of security incidents is an opaque process that involves the collection and correlation of diverse evidence. In this work we first conduct a complex experiment to expand our understanding of forensics analysis processes. During a period of 4 weeks, we systematically investigated 200 detected security incidents about compromised hosts within a large operational network. We used data from four commonly used security sources, namely Snort alerts, reconnaissance and vulnerability scanners, blacklists, and a search engine, to manually investigate these incidents. Based on our experiment, we first evaluate the (complementary) utility of the four security data sources and surprisingly find that the search engine provided useful evidence for diagnosing many more incidents than more traditional security sources, i.e., blacklists, reconnaissance, and vulnerability reports. Based on our validation, we then identify and make publicly available a list of 165 good Snort signatures, i.e., signatures that were effective in identifying validated malware without producing false positives. In addition, we analyze the characteristics of good signatures and identify strong correlations between different signature features and their effectiveness, i.e., the number of validated incidents in which a good signature is identified. Based on our experiment, we finally introduce an IDS signature quality metric that can be exploited by security specialists to evaluate the available rulesets, prioritize the generated alerts, and facilitate the forensics analysis processes. We apply our metric to characterize the most popular Snort rulesets. Our analysis of signatures is useful not only for configuring Snort but also for establishing best practices and for teaching how to write new IDS signatures.

Value Sensitive Transfer (VST) of Systems Among Countries

Systems like large technical and operational networks are necessary in modern societies, yet they are costly and time-consuming to develop. Instead of countries and organizations having to build systems from scratch, the transfer of systems is becoming more common. Yet systems reflect the values of the societies in which they are built and of the designers who develop them. Public values differ among cultures and countries; this not only hinders the transfer of systems but results in a lack of their adoption and acceptance by the receiving country. This article investigates the case of the transfer of parliamentary webcasting/telecasting systems from the US to Pakistan to better understand the international transfer of e-government systems. Although the concept of systems transfer is simple, implementing the system within a different cultural setting was more complicated than initially anticipated. The transfer of the system was influenced by the political objectives and cultural differences. Value tensions were found, especially surrounding openness, transparency, and accountability. Hence, the authors propose broadening the perspective on the transfer and development of systems by taking value differences into consideration. Toward this purpose, a framework for designing Value Sensitive Transfers (VST) is proposed.

An Efficient Formal Framework for Intrusion Detection Systems

Traffic anomalies and attacks are commonplace in today's networks, and identifying them rapidly and accurately is critical for large network operators. Intrusion detection systems are an important component of defensive measures protecting computer systems and networks from abuse.For an intrusion detection system, it is important to detect previously known attacks with high accuracy. However, detecting previously unseen attacks is equally important in order to minimize the losses as a result of a successful intrusion. It is also equally important to detect attacks at an early stage in order to minimize their impact. To address these challenges, this paper proposes to improve the efficiency of the network intrusion detection process by including an Event Calculus based specification to detect the registered and expected behaviour of the whole network.

Scalable monitoring via threshold compression in a large operational 3G network

Threshold-based performance monitoring in large 3G networks is very challenging for two main factors: large network scale and dynamics in both time and spatial domains. There exists a fundamental tradeoff between the size of threshold settings and the alarm quality. In this paper, we propose a scalable monitoring solution, called threshold-compression that characterizes the tradeoff via intelligent threshold aggregation. The main insight behind our solution is to identify groups of network elements with similar threshold behaviors across location and time dimensions, thus forming spatial-temporal clusters and generating the associated compressed thresholds within the optimization framework. Our evaluations on a commercial 3G network have demonstrated the effectiveness of our threshold-compression solution, e.g., threshold setting reduction up to 90% within 10% false/miss alarms.

Modeling a policy-capable path-vector routing protocol using Jacobi iteration over a path algebra

Several computer network design and analysis activities either require or benefit from having knowledge of the sets of paths possible between the network’s nodes. Often these activities utilize path sets containing those network paths of shortest length due to the availability of shortest path algorithms. However, in large operational IP networks (including the whole public Internet), user data may be forwarded along network paths which are not of shortest length due to the effects of inter-domain routing policies. To determine these policy-influenced network paths, two path algebras are developed which model a baseline and enhanced policy-capable path-vector routing protocol respectively. The enhanced model extends the capability of the baseline by including support for local preferencing and selective export filtering of network paths to support common Internet routing policies. Although both new path algebras can be considered “weak” in that they lack a few specific algebraic properties (such as being right distributive over multiplication), it is shown that both models can use Jacobi iteration to converge to sets of paths that form a (routing) in-tree. These sets of network paths are equivalent to that output by a path-vector routing protocol operating over a user-supplied topology. For large network topologies, the baseline routing protocol model is shown to require an order of magnitude less run-time and memory to calculate the same path sets as the de facto approach of discrete-event network simulation.

Detecting the performance impact of upgrades in large operational networks

Networks continue to change to support new applications, improve reliability and performance and reduce the operational cost. The changes are made to the network in the form of upgrades such as software or hardware upgrades, new network or service features and network configuration changes. It is crucial to monitor the network when upgrades are made because they can have a significant impact on network performance and if not monitored may lead to unexpected consequences in operational networks. This can be achieved manually for a small number of devices, but does not scale to large networks with hundreds or thousands of routers and extremely large number of different upgrades made on a regular basis. In this paper, we design and implement a novel infrastructure MERCURY for detecting the impact of network upgrades (or triggers) on performance. MERCURY extracts interesting triggers from a large number of network maintenance activities. It then identifies behavior changes in network performance caused by the triggers. It uses statistical rule mining and network configuration to identify commonality across the behavior changes. We systematically evaluate MERCURY using data collected at a large tier-1 ISP network. By comparing to operational practice, we show that MERCURY is able to capture the interesting triggers and behavior changes induced by the triggers. In some cases, MERCURY also discovers previously unknown network behaviors demonstrating the effectiveness in identifying network conditions flying under the radar.

Wavelet based Denial-of-Service detection

Network Denial-of-Service (DoS) attacks that disable network services by flooding them with spurious packets are on the rise. Criminals with large networks (botnets) of compromised nodes (zombies) use the threat of DoS attacks to extort legitimate companies. To fight these threats and ensure network reliability, early detection of these attacks is critical. Many methods have been developed with limited success to date. This paper presents an approach that identifies change points in the time series of network packet arrival rates. The proposed process has two stages: (i) statistical analysis that finds the rate of increase of network traffic, and (ii) wavelet analysis of the network statistics that quickly detects the sudden increases in packet arrival rates characteristic of botnet attacks. Most intrusion detections are tested using data sets from special security testing configurations, which leads to unacceptable false positive rates being found when they are used in the real world. We test our approach using data from both network simulations and a large operational network. The true and false positive detection rates are determined for both data sets, and receiver operating curves use these rates to find optimal parameters for our approach. Evaluation using operational data proves the effectiveness of our approach.

Estimating point-to-point and point-to-multipoint traffic matrices: an information-theoretic approach

Traffic matrices are required inputs for many IP network management tasks, such as capacity planning, traffic engineering, and network reliability analysis. However, it is difficult to measure these matrices directly in large operational IP networks, so there has been recent interest in inferring traffic matrices from link measurements and other more easily measured data. Typically, this inference problem is ill-posed, as it involves significantly more unknowns than data. Experience in many scientific and engineering fields has shown that it is essential to approach such ill-posed problems via regularization. This paper presents a new approach to traffic matrix estimation using a regularization based on entropy penalization. Our solution chooses the traffic matrix consistent with the measured data that is information-theoretically closest to a model in which source/destination pairs are stochastically independent. It applies to both point-to-point and point-to-multipoint traffic matrix estimation. We use fast algorithms based on modern convex optimization theory to solve for our traffic matrices. We evaluate our algorithm with real backbone traffic and routing data, and demonstrate that it is fast, accurate, robust, and flexible.

Network and domain autoconfiguration: a unified approach for large dynamic networks

Configuration management is critical to correct and efficient operation of large networks. Where the users and networks are dynamic and ad hoc, manual configuration quickly becomes too complex. The combination of the sheer number of nodes with heterogeneity and dynamics makes it almost impossible for the system administrator to ensure good configuration or even correct operation. To achieve the vision of pervasive computing, nodes must automatically discover their environment and self-configure, then automatically reconfigure to adapt to changes. Protocols such as DHCP, DDNS, and mDNS provide some degree of host autoconfiguration, but network administrators must still configure information such as address pools, routing protocols, and OSPF routing areas. Only limited progress has been made in automating the configuration of routers, servers, and network topology. We propose the first unified attempt to combine both self-configuration of much of the host, router, and server information with automatic generation and maintenance of hierarchy under the same algorithmic framework. Testbed implementations show the approach is practical, while simulations reveal its scalability, rapidity, and efficiency with respect to network performance.

Fast accurate computation of large-scale IP traffic matrices from link loads

A matrix giving the traffic volumes between origin and destination in a network has tremendously potential utility for network capacity planning and management. Unfortunately, traffic matrices are generally unavailable in large operational IP networks. On the other hand, link load measurements are readily available in IP networks. In this paper, we propose a new method for practical and rapid inference of traffic matrices in IP networks from link load measurements, augmented by readily available network and routing configuration information. We apply and validate the method by computing backbone-router to backbone-router traffic matrices on a large operational tier-1 IP network -- a problem an order of magnitude larger than any other comparable method has tackled. The results show that the method is remarkably fast and accurate, delivering the traffic matrix in under five seconds.