Failure Recovery Time Research Articles

High-Reliability Sub-Nanosecond Network Time Synchronization Method Enabled by Double-Frequency Distributed Time Synchronization

Time synchronization is a long-standing challenge in distributed systems like indoor/outdoor positioning, coordinated multi-point in 4G/5G mobile communication, etc., which require nanosecond-level high-reliable time sync networks. At present, a widely adopted solution for time sync networks is the precision time protocol specified in IEEE 1588v2, which can provide sub-microsecond sync accuracy. In addition, a novel network time sync method called distributed time synchronization has also been proposed recently, which can achieve 50-ns-level accuracy and stronger survivability in metro, regional, and backbone networks. However, both of the above methods encounter difficulties in achieving nanosecond sync accuracy and network reliability simultaneously. In this paper, by analyzing the main factors influencing the degradation of time sync accuracy, we reveal that the finite clock resolution is a major barrier to achieving nanosecond-level time synchronization. In order to establish a low-cost, high-reliability, and sub-nanosecond-level time sync network, we propose a novel network time sync method called double-frequency distributed time synchronization (DF- DTS). By selecting two specific and distinct frequencies for the sending and receiving clocks of the nodes/devices being synchronized, the time sync errors induced by the finite clock resolution can be reduced by a statistical approach. We set up a theoretical model for the DF-DTS and analyze the time sync accuracy through both mathematical derivations and network simulations. Furthermore, we propose a failure-restoration mechanism to enhance the reliability of DF-DTS networks by improving the time sync accuracy under failures and reducing the failure recovery time. Finally, we conduct both point- to-point and network time sync experiments to validate the proposed DF-DTS method. The results demonstrate that DF-DTS can achieve sub-nanosecond-level sync accuracy that is 1-2 orders of magnitude higher than the clock resolution in a prototype four-node DF-DTS network. Moreover, a network simulation under failure cases is conducted, and the results show that our method has significant advantages in both time sync accuracy and recovery time in the failure-restoration process compared to IEEE 1588v2.

Refactoring Network Functions Modules to Reduce Latencies and Improve Fault Tolerance in NFV

Network functions virtualization (NFV) allows service providers to deliver new services to their customers more quickly by adopting software-centric network functions implementation over commercial, off-the-shelf hardwares. This NFV-based software-centric approach cannot use dedicated mechanisms implemented over custom built boxes to reduce latencies and tolerate faults. We present a case study of IP multimedia subsystem (IMS), which is the most complex NFV instance, requires extremely low end-to-end latency (40 msec), and demands system availability as high as five nines. Through an empirical study, we discover that highly modular IMS network functions implementation over virtualized platform: 1) incurs latencies and 2) does not tolerate faults. NFV-based IMS modules incur high latencies by creating a feedback loop among each other while executing delay sensitive data-plane traffic. These IMS modules are also susceptible to failure, causing the control-plane to terminate the application session while keeping the data-plane to forward data packets. To address these issues, we propose to refactor network function modules. We reduce latencies by pipelining the IMS modules, and recover failed modules by reconfiguring their neighboring modules. We build our system prototype of open source IMS over OpenStack platform. Our results show that our scheme reduces latencies and failure recovery time up to $12\times $ and $10\times $ , respectively, when compared with the state-of-the-art virtualized IMS implementation.

Realizing Highly-Available, Scalable, and Protocol-Independent vSDN Slicing With a Distributed Network Hypervisor System

In this paper, we design and implement a distributed network virtualization hypervisor (NVH) system, namely, DPVisor, which can provide superior network programmability based on protocol-oblivious forwarding to realize highly-available, scalable, and protocol-independent virtual software-defined network slicing. The experimental comparisons indicate that DPVisor achieves comparable performance as ONVisor ( i.e. , an OpenFlow-based ONOS benchmark), in terms of the message processing latency, message processing throughput, and failure recovery time. Moreover, to optimize the performance of the distributed NVH systems, we carefully adjust the consistency model used in the state synchronization of NVH instances and propose a local cache-based scheme to balance the tradeoff between the consistency and availability of network status. Our experimental results confirm that the message processing throughput of the distributed NVH systems can be greatly improved ( i.e. , for both DPVisor and ONVisor), while the data consistency among NVH instances is still maintained well.

An auditing language for preventing correlated failures in the cloud

Today's cloud services extensively rely on replication techniques to ensure availability and reliability. In complex datacenter network architectures, however, seemingly independent replica servers may inadvertently share deep dependencies (e.g., aggregation switches). Such unexpected common dependencies may potentially result in correlated failures across the entire replication deployments, invalidating the efforts. Although existing cloud management and diagnosis tools have been able to offer post-failure forensics, they, nevertheless, typically lead to quite prolonged failure recovery time in the cloud-scale systems. In this paper, we propose a novel language framework, named RepAudit, that manages to prevent correlated failure risks before service outages occur, by allowing cloud administrators to proactively audit the replication deployments of interest. In particular, RepAudit consists of three new components: 1) a declarative domain-specific language, RAL, for cloud administrators to write auditing programs expressing diverse auditing tasks; 2) a high-performance RAL auditing engine that generates the auditing results by accurately and efficiently analyzing the underlying structures of the target replication deployments; and 3) an RAL-code generator that can automatically produce complex RAL programs based on easily written specifications. Our evaluation result shows that RepAudit uses 80x less lines of code than state-of-the-art efforts in expressing the auditing task of determining the top-20 critical correlated-failure root causes. To the best of our knowledge, RepAudit is the first effort capable of simultaneously offering expressive, accurate and efficient correlated failure auditing to the cloud-scale replication systems.

Congestion-Aware Local Reroute for Fast Failure Recovery in Software-Defined Networks

Although a restoration approach derives a reroute path when failure occurs and greatly reduces forwarding rules in switches compared with a protection approach, software-defined networks (SDNs) induce a long failure recovery process because of frequent flow operations between the SDN controller and switches. Accordingly, it is indispensable to design a new resilience approach to balance failure recovery time and forwarding rule occupation. To this end, we leverage flexible flow aggregation in fast reroute to solve this problem. In the proposed approach, each disrupted traffic flow is reassigned to a local reroute path for the purpose of congestion avoidance. Thus, all traffic flows assigned to the same local reroute path are aggregated into a new “big” flow, and the number of reconfigured forwarding rules in the restoration process is greatly reduced. We first formulate this problem as an integer linear programming model, then design an efficient heuristic named the “congestion-aware local fast reroute” (CALFR). Extensive emulation results show that CALFR enables fast recovery while avoiding link congestion in the post-recovery network.

$\mathrm {F^{2}}$ Tree: Rapid Failure Recovery for Routing in Production Data Center Networks

Failures are not uncommon in production data center networks (DCNs) nowadays. It takes long time for the DCN routing to recover from a failure and find new forwarding paths, significantly impacting realtime and interactive applications at the upper layer. In this paper, we present a fault-tolerant DCN solution, called ${\mathrm {F^{2}}}$ Tree, which is readily deployed in existing DNCs. ${\mathrm {F^{2}}}$ Tree can significantly improve the failure recovery time only through a small amount of link rewiring and switch configuration changes. Through testbed and emulation experiments, we show that ${\mathrm {F^{2}}}$ Tree can greatly reduce the routing recovery time after failure (by 78%) and improve the performance of upper layer applications when routing failure happens (96% less deadline-missing requests).

On Fault Tolerance for Distributed Iterative Dataflow Processing

Large-scale graph and machine learning analytics widely employ distributed iterative processing. Typically, these analytics are a part of a comprehensive workflow, which includes data preparation, model building, and model evaluation. General-purpose distributed dataflow frameworks execute all steps of such workflows holistically. This holistic view enables these systems to reason about and automatically optimize the entire pipeline. Here, graph and machine learning analytics are known to incur a long runtime since they require multiple passes over the data until convergence is reached. Thus, fault tolerance and a fast-recovery from any intermittent failure is critical for efficient analysis. In this paper, we propose novel fault-tolerant mechanisms for graph and machine learning analytics that run on distributed dataflow systems. We seek to reduce checkpointing costs and shorten failure recovery times. For graph processing, rather than writing checkpoints that block downstream operators, our mechanism writes checkpoints in an unblocking manner that does not break pipelined tasks. In contrast to the conventional approach for unblocking checkpointing (e.g., that manage checkpoints independently for immutable datasets), we inject the checkpoints of mutable datasets into the iterative dataflow itself. Hence, our mechanism is iteration-aware by design. This simplifies the system architecture and facilitates coordinating checkpoint creation during iterative graph processing. Moreover, we are able to rapidly rebound, via confined recovery, by exploiting the fact that log files exist locally on healthy nodes and managing to avoid a complete recomputation from scratch. In addition, we propose replica recovery for machine learning algorithms, whereby we employ a broadcast variable that enables us to quickly recover without having to introduce any checkpoints. In order to evaluate our fault tolerance strategies, we conduct both a theoretical study and experimental analyses using Apache Flink and discover that they outperform blocking checkpointing and complete recovery.

Network Intelligence Based on Network State Information for Connected Vehicles Utilizing Fog Computing

This paper proposes a method to take advantage of fog computing and SDN in the connected vehicle environment, where communication channels are unstable and the topology changes frequently. A controller knows the current state of the network by maintaining the most recent network topology. Of all the information collected by the controller in the mobile environment, node mobility information is particularly important. Thus, we divide nodes into three classes according to their mobility types and use their related attributes to efficiently manage the mobile connections. Our approach utilizes mobility information to reduce control message overhead by adjusting the period of beacon messages and to support efficient failure recovery. One is to recover the connection failures using only mobility information, and the other is to suggest a real-time scheduling algorithm to recover the services for the vehicles that lost connection in the case of a fog server failure. A real-time scheduling method is first described and then evaluated. The results show that our scheme is effective in the connected vehicle environment. We then demonstrate the reduction of control overhead and the connection recovery by using a network simulator. The simulation results show that control message overhead and failure recovery time are decreased by approximately 55% and 5%, respectively.

Fog Computing을 적용한 Connected Vehicle 환경에서 상태 정보에 기반한 네트워크 지능화

This paper proposes a method to take advantage of fog computing and SDN in the connected vehicle environment, where communication channels are unstable and the topology changes frequently. A controller knows the current state of the network by maintaining the most recent network topology. Of all the information collected by the controller in the mobile environment, node mobility information is particularly important. Thus, we divide nodes into three classes according to their mobility types and use their related attributes to efficiently manage the mobile connections. Our approach utilizes mobility information to reduce control message overhead by adjusting the period of beacon messages and to support efficient failure recovery. One is to recover the connection failures using only mobility information, and the other is to suggest a real-time scheduling algorithm to recover the services for the vehicles that lost connection in the case of a fog server failure. A real-time scheduling method is first described and then evaluated. The results show that our scheme is effective in the connected vehicle environment. We then demonstrate the reduction of control overhead and the connection recovery by using a network simulator. The simulation results show that control message overhead and failure recovery time are decreased by approximately 55% and 5%, respectively.

Multi-Domain SDN Survivability for Agricultural Wireless Sensor Networks.

Wireless sensor networks (WSNs) have been widely applied in agriculture field; meanwhile, the advent of multi-domain software-defined networks (SDNs) have improved the wireless resource utilization rate and strengthened network management. In recent times, multi-domain SDNs have been applied to agricultural sensor networks, namely multi-domain software-defined wireless sensor networks (SDWSNs). However, when the SDNs controlling agriculture networks suddenly become unavailable, whether intra-domain or inter-domain, sensor network communication is abnormal because of the loss of control. Moreover, there are controller and switch info-updating problems even if the controller becomes available again. To resolve these problems, this paper proposes a new approach based on an Open vSwitch extension for multi-domain SDWSNs, which can enhance agriculture network survivability and stability. We achieved this by designing a connection-state mechanism, a communication mechanism on both L2 and L3, and an info-updating mechanism based on Open vSwitch. The experimental results show that, whether it is agricultural inter-domain or intra-domain during the controller failure period, the sensor switches can enter failure recovery mode as soon as possible so that the sensor network keeps a stable throughput, a short failure recovery time below 300 ms, and low packet loss. Further, the domain can smoothly control the domain network again once the controller becomes available. This approach based on an Open vSwitch extension can enhance the survivability and stability of multi-domain SDWSNs in precision agriculture.

CDNPatch: a cost‐effective failover mechanism for hybrid CDN‐P2P live streaming systems

SummaryAmong the most well‐established live media distribution technologies is content delivery network (CDN), which improves user‐perceived quality of service by delivering content from proxy servers deployed at the Internet's edge. In recent years, CDN providers started to tap into their subscribers' peer‐to‐peer (P2P) capacity to alleviate their server costs. Under the inherent peer dynamics, a major challenge of these hybrid CDN‐P2P systems is to provide efficient failure recovery with good quality of service guarantees at a reduced server cost. In this work we propose a cost‐effective failover solution named CDNPatch to address the aforementioned problem. CDNPatch enables peers to periodically precompute a few backup content suppliers by efficient information exchange and maintenance algorithms, and leverages auxiliary CDN servers and an economic server provisioning algorithm to reduce the chance of playback interruption occurring to peers. Our simulation results show that CDNPatch can mask the impact of peer dynamics of 3 real P2P systems, namely, SOPCast, PPStream, and PPTV, with 100 % failure recovery success rate and a failure recovery time less than 1 second at a cost of small P2P communication overhead of less than 1 kilobits per second, while using only 10%, 21%, and 51%, respectively, of the pure CDN scheme's server consumption.

Incremento en la fiabilidad de un enlace táctico naval mediante el diseño y la implementación de mecanismos de recuperación automática ante fallas

This document describes the design and implementation process of an automatic failure recovery system on a tactical data link, with the purpose of increasing its reliability during the execution of command and control naval operations. is effort is part of a project that seeks to create a Command and Control System at an operational level. The design of the recovery mechanism begins with the state analysis of the data link system under study, it then, continues with the identification of possible “dead states” and finally, turns to the development of software solutions for each identified failure. is solution considers the communication infrastructure capabilities currently available in the Navy units, so its implementation costs are reduced. is paper also presents the results obtained from the tests carried out in the system, which show that the average failure recovery time was reduced by 50%, increasing the reliability of the analyzed data link.

Cost of Loop-Free Alternates in IP-Over-WDM Networks

In Internet protocol (IP) networks, the failure recovery time of IP restoration, of several hundreds of milliseconds or up to seconds, is sufficient for most applications. But protection switching is needed for services that require very short recovery times of at most 50 ms. Loop-free alternates (LFAs) is a protection switching mechanism that is in the IP layer. We consider IP-over-WDM (wavelength division multiplexed) networks where IP routing eventually restores all traffic, while LFAs protect the fraction of the traffic that has high priority. Simulations show that if 50% of the traffic has high priority, then the additional cost to protect the high-priority traffic using LFAs is just a percent on average, under a particular cost model. Thus, with very little additional network cost, LFAs can guarantee that a large fraction of traffic will be protected between all source-destination pairs. In addition, a network design algorithm is presented that, given a traffic matrix and a physical network topology, will find a low-cost network design that includes determining the IP network topology, IP-link capacities, lightpath routing, and LFAs.

A Proposal of Cyclic Sleep Control Technique for Backup Resources in ROADM Systems to Reduce Power Consumption of Photonic Network

We propose a cyclic sleep control technique for backup resources in reconfigurable optical add/drop multiplexer (ROADM) systems to simultaneously achieve power savings and high-speed recovery from failures. Processes to check the reliability of backup resources, backup transponders and paths, are also provided in the control technique. The proposed technique uses sleep mode where backup transponders are powered down to minimize power for power savings. At least one of the backup transponders is always activated after self-checking using the loopback fiber connection in the ROADM and it becomes a shared backup for working transponders to enable high-speed recovery from failures. This activated backup transponder is powered down again after the next transponder is activated. These state transitions are cyclically applied to each backup transponder. This “cyclic” aspect of operation enables network operators to continuously monitor the reliability for all backup resources with the sleep mode. The activated backup transponders at both ends of the path are used in checking the reliability of backup paths. Therefore, all backup resources, both transponders and paths, can be regularly checked with the sleep mode to ensure data are stably forwarded. We estimated the power consumption with this technique under various conditions and found a trade-off between power reduction and the recovery capabilities from failures. We achieved more than 34% power saving of backup transponders maintaining the failure recovery time within 50ms in experiments. Furthermore, we confirmed the reliability of backup paths in experiments using backup transponders with the cyclic sleep control technique. These results indicated that the proposed control technique is promising in dramatically and reliably reducing the power consumption of backup resources.

NetPilot

Driven by the soaring demands for always-on and fast-response online services, modern datacenter networks have recently undergone tremendous growth. These networks often rely on commodity hardware to reach immense scale while keeping capital expenses under check. The downside is that commodity devices are prone to failures, raising a formidable challenge for network operators to promptly handle these failures with minimal disruptions to the hosted services. Recent research efforts have focused on automatic failure localization. Yet, resolving failures still requires significant human interventions, resulting in prolonged failure recovery time. Unlike previous work, NetPilot aims to quickly mitigate rather than resolve failures. NetPilot mitigates failures in much the same way operators do -- by deactivating or restarting suspected offending components. NetPilot circumvents the need for knowing the exact root cause of a failure by taking an intelligent trial-and-error approach. The core of NetPilot is comprised of an Impact Estimator that helps guard against overly disruptive mitigation actions and a failure-specific mitigation planner that minimizes the number of trials. We demonstrate that NetPilot can effectively mitigate several types of critical failures commonly encountered in production datacenter networks.

Distributed Fault-Tolerant Quality of Wireless Networks

A mobile ad hoc network (MANET) consists of a group of communicating hosts that form an arbitrary network topology by means of any of several wireless communication media. MANET communications represent a diversification in communication technology necessary to solve the stringent end-to-end requirements of QoS-based communication networks. Of the many challenges in this complex distributed system, the problem of routing based on a predefined set of customer preferences, critical to guaranteeing quality-of-service, is the focus of this research. Specifically, this paper modifies a cluster-based QoS routing algorithm for mobile ad hoc networks with the aim of providing fault tolerance, which is a critical feature in providing QoS in the link failure-prone environment of mobile networks. Performance of this new fault-tolerant cluster-based QoS wireless algorithm is evaluated according to failure recovery time, dropped packets, throughput, and sustained flow bandwidth via simulations involving node failure scenarios along QoS paths.

Design and Implementation of High Availability OSPF Router

In this paper, we propose a High Availability Open Shortest Path First (HA-OSPF) router which consists of two OSPF router modules, active and standby, to support a high availability network. First, we used the continuous-time Markov chain (CTMC) to analyze the steady-state availability of an HA-OSPF router with one active router and N standby routers (1 + N redundancy model). Then, with the failure detection and recovery rate considered, from analytic results, we show that the HA-OSPF router with 1 + 1 redundancy model, one active and one standby, is the preferred model for enhancing router availability. We also show that the carrier-grade HA-OSPF router availability (i.e., five-nine availability) can be achieved under an appropriate combination of the router module failure rate (λ), repair rate (μ), and the failure detection and recovery rate (δ). Since there is a lack of research on the integration of the redundancy model, link state information backup, and failure detection and recovery, we propose a high availability management middleware (HAM middleware) framework to integrate these three elements. The HAM middleware consists of Availability Management Framework (AMF) service, Checkpoint service, and Failure Manager. It supports health check, state information exchange, and failure detection and recovery. Each HA-OSPF router was designed to have a Linux operating system, HAM middleware, and OSPF process. We have implemented the HA-OSPF router on a PC-based system. Experimental results show that the failure detection and recovery times of the proposed PC-based HA-OSPF router were reduced by 98.76% and 91.45% compared to those of an industry standard approach, VRRP (Virtual Router Redundancy Protocol), for a software failure and a hardware failure, respectively. In addition, we have also implemented the HA-OSPF router on an ATCA (Advanced Telecom Computing Architecture) platform, which can provide an industrial standardized modular architecture for an efficient, flexible, and reliable router design. Based on our ATCA-based platform with 1/δ=217 ms for a software failure and 1/δ=1066 ms for a hardware failure, along with the router module data, 1/λ = 7 years and 1/μ = 4 hours, obtained from Cisco, the availabilities of the proposed ATCA-based HA-OSPF router are 99.99999905% for a software failure and 99.99999867% for a hardware failure. Therefore, the experimental results have shown that both our proposed ATCA-based and PC-based HA-OSPF routers with 1 + 1 redundancy model can easily meet the requirement of carrier-grade availabilities with five-nine.

A High-Speed Protection Scheme for Multiple-Priority-Class Traffic in WDM Ring Networks

We propose a high-speed protection scheme for multiple-priority-class traffic transmission in WDM ring networks. This scheme achieves high-speed protection by quickly suspending transmission of low-priority traffic when a failure is detected. Each node suspends transmission of the low-priority traffic being sent over the backup path corresponding to the impaired primary path after receiving a failure notification and checking a table that includes the primary paths passing through the node, the backup paths corresponding to the primary paths, and low-priority traffic transmitted by the node. If a node detects a failure, it sends a single failure notification for each ring to the source node of the primary path, and the nodes on the route sequentially suspend low-priority traffic. Simulation shows that the proposed scheme reduces the failure-recovery time by up to 60% compared with the conventional scheme.

Survivable and traffic-oblivious routing in WDM networks: valiant load balancing versus tree routing

This paper investigates the issue of survivable routing to prevent arbitrary single-link failures in WDM networks when the traffic information is partially known. Two novel protection schemes called valiant-load-balancing- (VLB-) based hop constraint segment protection (VLB-HCSP) and hop constraint minimizing leaf nodes (HCMLN) are proposed and evaluated. Simulation results confirm our expectation that VLB-HCSP can achieve a desirable compromise between cost and failure recovery time compared with the previous VLB-based method, while HCMLN performs even more inspiringly for being able to provide both a low-cost budget and a fast recovery at the same time, in contrast to VLB-HCSP and the other existing schemes.

A proactive backup scheme for reliable real-time transmission

Reliable transmission is a key issue for distributed real-time applications. The concept of Real-time Dependable Channel was introduced to provide availability to real-time transmission. Two aspects are important for the efficiency of a Real-time dependable channel: assuring the end-to-end delay bound and optimising the utilisation of network resources. A packet can miss its delay bound for two reasons: network congestion or network failure. The classic solution to this problem has been the use of Backup Channels which introduces the notion of availability at the expense of increasing the use of network resources. However, this over-provisioning of resources is potentially wasted, since the failure rate is very low. This paper introduces a new failure detection scheme for Real-time Transmission, which is called Proactive Backup Channel. This scheme is based on activating the backup channel before a network failure or congestion is produced. This way, the failure recovery time is reduced, and, as proven in the paper, the use of network resources is minimized.