Fast Failure Recovery Research Articles

Fast Fail-Over Technique for Distributed Controller Architecture in Software-Defined Networks

Recent studies have proposed different approaches to mitigate the risk of overload and failure in Software-Defined Networks (SDNs). Some of these approaches have proven effective but only in specific use cases, making it potentially difficult to generalize their application. Furthermore, network failure detection and recovery by the SDN control plane requires sophisticated software logic running on multiple controllers that are non-intrusive to the network environment. While this allows more flexibility to respond to failure events, it also implies that each controller application must include its recovery logic, which increases code complexity. In this paper, we propose a fast fail-over technique for solving the problem of a controller failure or target availability in the network. We argue that inter-domain controller synchronization can result in high network overhead and should be minimized to per-need base only. To this end, upon detecting a failure in the control plane, the proposed fast failure recovery technique leverages a load-shifting scheme to initialize alternate paths and proactively instantiate flow rules to reduce flow setup latency. To prevent packet loss during failure recovery, we utilize a forwarding information table that quickly replays inputs to the controller after failure recovery. Our extensive experiments show that the average latency incurred by the controller to controller communication is approximately twice that of per-need based synchronization. The experimental results also show that our proposed technique achieved a 50% reduction in service interruption period and 75% flow_mod reduction during a single link failure over the traditional SDN baseline approach.

Software-Defined Networking Switches for Fast Single-Link Failure Recovery

In this paper, we consider IP fast recovery from single-link failures in a given network topology. The basic idea is to replace some existing routers with a designated switch. When a link fails, the affected router will send all the affected traffic to the designated switch (through pre-configured IP tunnels), which will deliver the affected traffic to its destination without using the failed link. The goal of the approach is to achieve faster failure recovery than traditional routing protocols that employ reactive computing upon link failures. Software-Defined Networking (SDN) switches can serve as the designated switches because they can flexibly redirect affected traffic to other routes, instead of only to the shortest paths in the network. However, SDN switches are very expensive. Our objective is to minimize the number of SDN switches needed and to guarantee that the network can still recover from any single-link failure. For networks with uniform link costs, we show that using normal non-SDN switches with IP tunneling capability as designated switches can guarantee recovery from any single-link failure. For networks with general link costs, we find that not all single-link failures can be recovered by using non-SDN switches as designated switches; by using SDN switches only when necessary, we can reduce the total number of SDN switches needed compared to an existing work. We conduct extensive simulations to verify our proposed approaches.

State Management Function Placement for Service-Based 5G Mobile Core Architecture

Service-based 5G core architecture is designed to take advantages of network function virtualization and software-defined networking. In addition to the control and data plane separation, the service-based 5G core decouples the computing and storage resources which separates the mobile functions into two categories: “stateless” control functions and state management functions. Such new features improve 5G core network in terms of independent scalability and fast failure recovery. In geo-distributed cloud infrastructure-based 5G core networks, the “stateless” control functions can be deployed to all cloud centers close to access networks to reduce latency and traffic load burden. However, we can not deploy state management functions to all cloud centers which results in the high state transfer cost. In addition, we can not use only one state management function for entire network which results in high traffic load burden. Therefore, the placement of state management functions involves different conflicting design objectives which requires a new model to optimally place these functions. In this paper, we propose a multi-objective model which can achieve the balance between state transfer cost and traffic load burden on state management functions. We first solve our model using e − constraint approach which tries to optimize one objective while keeping another under threshold. Second, we propose an adaptive solution based on adaptive weighted sum approach to find a set of Pareto optimal solutions for our multi-objective model. Simulation results show that our proposed solution offers better balance between two design objectives compared to other solutions.

A Hierarchical RAID Architecture Towards Fast Recovery and High Reliability

Disk failures are very common in modern storage systems due to the large number of inexpensive disks. As a result, it takes a long time to recover a failed disk due to its large capacity and limited I/O. To speed up the recovery process and maintain a high system reliability, we propose a hierarchical code architecture with erasure codes, OI-RAID, which consists of two layers of codes, outer layer code and inner layer code. Specifically, the outer layer code is deployed with disk grouping technique based on Balanced Incomplete Block Design (BIBD) or complete graph with skewed data layout to provide efficient parallel I/O of all disks for fast failure recovery, and the inner layer code is deployed within each group of disks to provide high reliability. As an example, we deploy RAID5 in both layers to achieve fault tolerance of at least three disk failures, which meets the requirement of data availability in practical systems, as well as much higher speed up ratio for disk failure recovery than existing approaches. Besides, OI-RAID also keeps the optimal data update complexity and incurs low storage overhead in practice.

Link Failure Recovery in SDN: High Efficiency, Strong Scalability and Wide Applicability

Link failures are commonly observed in computer networks, including the newly emerging Software Defined Network (SDN). Considering that failure recovery methods used in traditional networks cannot be applied to SDN networks directly, we propose a method named pro-VLAN in this paper, which calculates a backup path and assigns a unique VLAN id for each link of the network based on the protection mechanism. It makes the most of SDN’s features and can recover a single link failure in SDN with the advantages of high efficiency, strong scalability and wide applicability. More specifically, high efficiency (i.e., a fast failure recovery with a low memory consumption) is achieved by calculating backup paths for each link instead of each flow and using group tables to switch backup paths automatically and locally when failures occur. Strong scalability (i.e., the amount of backup flow entries per switch is stable) is achieved by keeping the amount of links per switch no matter how the network scale extends or how the amount of flows increases. Wide applicability is achieved by always finding a path available without modifying any hardware or protocol as long as the network is still reachable after a link failure. Simulation results and mathematical analysis demonstrate that both pro-VLAN and a flow-based protection method achieve a fast failure recovery, while pro-VLAN consumes less than 1% of the forwarding entries to store backup paths as compared to the flow-based method. Moreover, when the network scale increases from 10 to 60 switches by 500%, the increase of the number of backup flow entries per switch built by pro-VLAN is only less than 50%.

Usage-Aware Protection Plan for State Management Functions in Service-Based 5G Core Network

In service-based 5G core networks, the mobile core functions can be decoupled as “stateless” control functions and state management functions to support fast failure recovery and independent scalability. With the arrival of the network function virtualization framework, these 5G mobile core functions can be deployed as virtual network functions over geo-distributed cloud infrastructures without much effort. With this new service-based 5G architecture, the availability of the core network depends on the availability of these state management functions. Therefore, protection plans are required to protect these state management functions. In this paper, we present a model to optimally place the standby functions protecting the active state management functions over geo-distributed cloud infrastructures. This model considers the constraints related to state management functions and considers a new factor (i.e., availability zones) in the estimation of the availability level of each protection plan. We focus on the optimization of total resources, including network bandwidth and computing resources. To minimize complexity, we propose a heuristic algorithm that achieves the optimal total resources while satisfying the availability requirements.

SDN Enabled Restoration With Triggered Precomputation in Elastic Optical Inter-Datacenter Networks

Flexi-grid elastic optical networks (EONs) provide a promising infrastructure for meeting the high-bandwidth and low-latency requirements of interconnecting datacenters. Since failures of high-capacity fiber links prevent users from accessing cloud services and lead to large amounts of data loss, much recent research has focused on inter-datacenter network failure recovery. In this paper, we build upon software-defined networking (SDN) technology and develop a fast restoration strategy with triggered precomputation (FR-TP) that achieves fast failure recovery with minimal resource overhead. Our work makes several contributions. We extend the controller functionality and OpenFlow protocol for software-defined elastic optical inter-datacenter network architectures, so as to obtain the global network state information quickly and accurately. We then present the FR-TP strategy and an associated trigger mechanism to compute backup paths before a link failure occurs. Furthermore, to improve the efficiency of path computation and bandwidth resource assignment, we construct a layered auxiliary graph of spectrum window planes (SWP-LAG) using a residual capacity matrix to change dynamically the width of the spectrum window planes (SWPs) to satisfy different service requests. Simulation results demonstrate that, compared with existing restoration strategies, the proposed FR-TP strategy combined with SWP-LAG reduces the recovery time by up to 30.4% in the network topologies studied without increasing the blocking probability.

Congestion Aware Fast Link Failure Recovery of SDN Network Based on Source Routing

The separation of control plane and data plane in Software Defined Network (SDN) makes it flexible to control the network behavior, while also causes some inconveniences to the link failure recovery due to the delay between fail point and the controller. To avoid delay and packet loss, pre-defined backup paths are used to reroute the disrupted flows when failure occurs. However, it may introduce large overhead to build and maintain these backup paths and is hard to dynamically construct backup paths according to the network status so as to avoid congestion during rerouting process. In order to realize congestion aware fast link failure recovery, this paper proposes a novel method which installs multi backup paths for every link via source routing and per-hop-tags and spread flows into different paths at fail point to avoid congestion. We carry out experiments and simulations to evaluate the performance of the method and the results demonstrate that our method can achieve congestion aware fast link failure recovery in SDN with a very low overhead.

Fast network restoration with equivalent cooperative routing

Aiming at the problem of failure recovery in current networks, a fast failure recovery method based on equivalent cooperative routing is proposed. Firstly, the transmission path between the source and destination nodes is divided into several non-overlapping path segments. Next, backup paths are deployed for each link in the path segment through segmented routing technology, which ensures fast routing recovery after failure. Additionally, in order to avoid damaging the QoS of the data stream through the failure recovery process, the transmission is guaranteed by the intersegment QoS complement. The experimental results show that the proposed method has a low failure recovery delay under a relatively small flow table cost.

CubicRing: Exploiting Network Proximity for Distributed In-Memory Key-Value Store

In-memory storage has the benefits of low I/O latency and high I/O throughput. Fast failure recovery is crucial for large-scale in-memory storage systems, bringing network-related challenges, including false detection due to transient network problems, traffic congestion during the recovery, and top-of-rack switch failures. In order to achieve fast failure recovery, in this paper, we present CubicRing, a distributed structure for cube-based networks, which exploits network proximity to restrict failure detection and recovery within the smallest possible one-hop range. We leverage the CubicRing structure to address the aforementioned challenges and design a network-aware in-memory key-value store called MemCube. In a 64-node 10GbE testbed, MemCube recovers 48 GB of data for a single server failure in 3.1 s. The 14 recovery servers achieve 123.9 Gb/s aggregate recovery throughput, which is 88.5% of the ideal aggregate bandwidth and several times faster than RAMCloud with the same configurations.

Fault tolerance in TCAM-limited software defined networks

In Software Defined Networks (SDNs), while a proactive fault tolerance based on the local rerouting approach enables fast failure recovery, it requires to install forwarding rules for the backup paths in the switch Ternary Content Addressable Memory (TCAM) in advance. Since the TCAM size is limited and forwarding rules are long, using large number of forwarding rules for backup paths leads to frequent eviction of flows at the switches. To guarantee the bandwidth requirement for a flow upon a link failure, bandwidth reservation is also required on the backup path. Inefficient backup bandwidth usage will cause rejection of increased number of flows. In this paper, we address the problem of proactive fault tolerance in SDNs, considering the above challenges by adopting the local rerouting approach. We develop an optimization programming formulation that determines the set of backup paths to protect a flow while minimizing the number of additional rules and bandwidth required for the backup paths. Since this problem is computationally prohibitive, we develop two heuristic algorithms, namely Forward Local Rerouting (FLR) and Backward Local Rerouting (BLR) to compute backup paths so as to improve TCAM and bandwidth usage efficiency. We propose a flexible adaptive failure recovery framework that takes the decision of using FLR or BLR based on the network state. We evaluate the proposed algorithms through simulations and compare their performance with the optimal solution. The results show that the proposed algorithms perform very close to the optimal solution and reduce the number of additional rules required to protect a flow by up to 75% compared to the approaches that do not consider the limited size of TCAM. The results also show that the proposed algorithms are effective in terms of backup bandwidth efficiency and meet the carrier-grade requirement with the recovery time below 50 ms.

Recursive SDN for Carrier Networks

Control planes for global carrier networks should be programmable and scalable. Neither traditional control planes nor new SDN-based control planes meet both of these goals. Here we propose a framework for recursive routing computations that combines the best of SDN (programmability through centralized controllers) and traditional networks (scalability through hierarchy) to achieve these two desired properties. Through simulation on graphs of up to 10,000 nodes, we evaluate our design's ability to support a variety of unicast routing and traffic engineering solutions, while incorporating a fast failure recovery mechanism based on network virtualization.

Fast failure detection and recovery in SDN with stateful data plane

SUMMARYWhen dealing with node or link failures in software‐defined networking (SDN), the network capability to establish an alternative path depends on controller reachability and on the round‐trip times between controller and involved switches. Moreover, current SDN data plane abstractions for failure detection, such as OpenFlow “Fast‐failover,” do not allow programmers to tweak switches' detection mechanism, thus leaving SDN operators relying on proprietary management interfaces (when available) to achieve guaranteed detection and recovery delays. We propose SPIDER, an OpenFlow‐like pipeline design that provides (i) a detection mechanism based on switches' periodic link probing and (ii) fast reroute of traffic flows even in the case of distant failures, regardless of controller availability. SPIDER is based on stateful data plane abstractions such as OpenState or P4, and it offers guaranteed short (few milliseconds or less) failure detection and recovery delays, with a configurable trade‐off between overhead and failover responsiveness. We present here the SPIDER pipeline design, behavioral model, and analysis on flow tables' memory impact. We also implemented and experimentally validated SPIDER using OpenState (an OpenFlow 1.3 extension for stateful packet processing) and P4, showing numerical results on its performance in terms of recovery latency and packet loss.

Enabling fast failure recovery in shared Hadoop clusters: Towards failure-aware scheduling

Hadoop emerged as the de facto state-of-the-art system for MapReduce-based data analytics. The reliability of Hadoop systems depends in part on how well they handle failures. Currently, Hadoop handles machine failures by re-executing all the tasks of the failed machines (i.e., executing recovery tasks). Unfortunately, this elegant solution is entirely entrusted to the core of Hadoop and hidden from Hadoop schedulers. The unawareness of failures therefore may prevent Hadoop schedulers from operating correctly towards meeting their objectives (e.g., fairness, job priority) and can significantly impact the performance of MapReduce applications. This paper presents Chronos, a failure-aware scheduling strategy that enables an early yet smart action for fast failure recovery while still operating within a specific scheduler objective. Upon failure detection, rather than waiting an uncertain amount of time to get resources for recovery tasks, Chronos leverages a lightweight preemption technique to carefully allocate these resources. In addition, Chronos considers data locality when scheduling recovery tasks to further improve the performance. We demonstrate the utility of Chronos by combining it with Fifo and Fair schedulers. The experimental results show that Chronos recovers to a correct scheduling behavior within a couple of seconds only and reduces the job completion times by up to 55% compared to state-of-the-art schedulers.

A Multicast Tree Management Method Supporting Fast Failure Recovery and Dynamic Group Membership Changes in OpenFlow Networks

A Multicast Tree Management Method Supporting Fast Failure Recovery and Dynamic Group Membership Changes in OpenFlow Networks

Integrated, Distributed Traffic Control in Multidomain Networks

In this paper, we put forward an integrated traffic control structure and the associated control laws for multidomain networks. This control structure performs per-edge-to-edge-based multinext-hop or multipath rate adaptation and load balancing among domain edge nodes in a multidomain network. This control structure is underpinned by a large family of distributed control laws, with provable convergence and optimality properties. With any user-defined global design objective, a set of control laws can be selected from this family of control laws that track an operational point where the global design objective is achieved, while providing traffic engineering (TE) and fast failure recovery (FFR) features for class-of-service (CoS)-aware flow aggregates. The structure allows the user to have full control over how the domains should be created and whether to use point-to-multipoint and/or point-to-point multipath. The flexibility and versatility of the control structure makes it an ideal theoretical underpinning for the development of integrated traffic control solutions for large-scale networking systems, in particular, software-defined networks in which the data plane is fully programmable via a well-defined south-bound interface, such as OpenFlow. The simulation testing demonstrates the viability of the solution in providing TE, FFR, and CoS features.

Fast reroute from single link and single node failures for IP multicast

The rise in multicast implementations has seen with it an increased support for fast failure recovery from link and node failures. Most recovery mechanisms augment additional services to existing protocols causing excessive overhead, and these modifications are predominantly protocol-specific. In this paper, we develop a multicast failure recovery mechanism that constructs protocol independent fast reroute paths to recover from single link and single node failures. We observe that single link failure recovery in multicast networks is similar to recovering unicast traffic, and we use existing unicast recovery mechanisms for multicast traffic. We construct multicast protection trees that provide instantaneous failure recovery from single node failures. For a given node x, the multicast protection tree spans all its neighbors and does not include itself. Thus, when the node fails, the neighbors of the node are connected through the multicast protection tree instead of node x, and forward the traffic over the multicast protection tree for the duration of failure recovery. The multicast protection trees are constructed a priori, without the knowledge of the multicast traffic in the network. Based on simulations on three realistic network topologies, we observe that the multicast protection trees increase the routing table size only by 38% on average and the path length between any source–destination pair by 13% on average.

Modeling and Algorithm for Multiple Spanning Tree Provisioning in Resilient and Load Balanced Ethernet Networks

We propose a multitree based fast failover scheme for Ethernet networks. In our system, only few spanning trees are used to carry working traffic in the normal state. As a failure happens, the nodes adjacent to the failure redirect traffic to the preplanned backup VLAN trees to realize fast failure recovery. In the proposed scheme, a new leaf constraint is enforced on the backup trees. It enables the network being able to provide 100% survivability against any single link and any single node failure. Besides fast failover, we also take load balancing into consideration. We model an Ethernet network as a twolayered graph and propose an Integer Linear Programming (ILP) formulation for the problem. We further propose a heuristic algorithm to provide solutions to large networks. The simulation results show that the proposed scheme can achieve high survivability while maintaining load balancing at the same time. In addition, we have implemented the proposed scheme in an FPGA system. The experimental results show that it takes only fewμsec to recover a network failure. This is far beyond the 50 msec requirement used in telecommunication networks for network protection.

Fast failure recovery in distributed graph processing systems

Distributed graph processing systems increasingly require many compute nodes to cope with the requirements imposed by contemporary graph-based Big Data applications. However, increasing the number of compute nodes increases the chance of node failures. Therefore, provisioning an efficient failure recovery strategy is critical for distributed graph processing systems. This paper proposes a novel recovery mechanism for distributed graph processing systems that parallelizes the recovery process. The key idea is to partition the part of the graph that is lost during a failure among a subset of the remaining nodes. To do so, we augment the existing checkpoint-based and log-based recovery schemes with a partitioning mechanism that is sensitive to the total computation and communication cost of the recovery process. Our implementation on top of the widely used Giraph system outperforms checkpoint-based recovery by up to 30x on a cluster of 40 compute nodes.

Optimizing large data transfers in parity-declustered data layouts

Disk arrays allow faster access to users' data by distributing the data among a collection of disks and allowing parallel access. Fault tolerance in a disk array can be achieved by using a data layout, and the technique of parity declustering allows faster failure recovery at the cost of additional space dedicated to redundant information. A collection of six performance conditions that parity-declustered data layouts should satisfy has guided most previous work; however two of these conditions (Maximal parallelism and Large write optimization) cannot be jointly satisfied in most cases. This limits the ability of parity-declustered data layouts to take full advantage of the available parallelism during large data transfers. We present data layouts that approximately satisfy these two conditions simultaneously for all possible array configurations, and bound the deviations from complete satisfaction. Our results yield improved performance guarantees for large data transfers in parity-declustered data layouts.