Failure Recovery Mechanism Research Articles

Proactive Failure Recovery for NFV in Distributed Edge Computing

Deploying NFV technologies to the edge networks has attracted growing attention in the stateof- the-art studies. In this article, we first review the most recent work on the topic of applying NFV to edge networks. Then, we identify that an urgent research challenge is to provide proactive failure recovery (shorten as failover hereafter) mechanism for the NFV-enabled distributed edge computing. To address this issue, we propose a novel management architecture that supports the proactive failover mechanism while provisioning NFV services in distributed edge computing. Simulation results show that the proposed proactive failover mechanism outperforms the reactive manner significantly in terms of latency spending on failover operations. We hope this article can spur deeper studies on the proactive intelligent resilience mechanism for deploying NFV in distributed edge computing and other related edge intelligence topics.

Failure Recovery in Robot–Human Object Handover

Object handover is a common physical interaction between humans. It is thus also of significant interest for human–robot interaction. In this paper, we are focused on robot-to-human object handover. The main challenge in this case is how to reduce the failure rate, i.e., to ensure that the object does not fall (object safety), while at the same time allowing the human to easily acquire the object (smoothness). To endow the robot with a failure recovery mechanism, we investigated how humans detect failure during the transfer phase of the handover. We conducted a human study that showed that a human giver primarily relies on vision rather than haptic sensing to detect the fall of the object. Motivated by this study, a robotic handover system is proposed that consists of a motion sensor attached to the robot's gripper, a force sensor at the base of the gripper, and a controller that is capable of regrasping the object if it starts falling. The proposed system is implemented on a Baxter robot and is shown to achieve a smooth and safe handover.

Dynamic multiple node failure recovery in distributed storage systems

Our daily lives are getting more and more dependent on data centers and distributed storage systems in general, whether at the business or at the personal level. With the advent of fog computing, personal mobile devices in a given geographical area may also comprise a very dynamic distributed storage system. These paradigm changes call for the urgent need of devising efficient and reliable failure recovery mechanisms in dynamic scenarios where failures become more likely and nodes join and leave the network more frequently. Redundancy schemes in distributed storage systems have become essential for providing reliability given the fact of frequent node failures. In this work, we address the problem of multiple failure recovery with dynamic scenarios using the fractional repetition code as a redundancy scheme. The fractional repetition (FR) code is a class of regenerating codes that concatenates a maximum distance separable code (MDS) with an inner fractional repetition code where data is split into several blocks then replicated and multiple replicas of each block are stored on various system nodes. We formulate the problem as an integer linear programming problem and extend it to account for three dynamic scenarios of newly arriving blocks, nodes, and variable priority blocks allocation. The contribution of this paper is four-fold: i. we generate an optimized block distribution scheme that minimizes the total system repair cost of all dependent and independent multiple node failure scenarios; ii. we address the practical scenario of having newly arriving blocks and allocate those blocks to existing nodes without any modification to the original on-node block distribution; iii. we consider new-comer nodes and generate an updated optimized block distribution; iv. we consider optimized storage and recovery of blocks with varying priority using variable fractional repetition codes. The four problems are modeled using incidence matrices and solved heuristically. We present a range of results for our proposed algorithms in several scenarios to assess the effectiveness of the solution approaches that are shown to generate results close to optimal.

Task Scheduling in Big Data Platforms: A Systematic Literature Review

Context: Hadoop, Spark, Storm, and Mesos are very well known frameworks in both research and industrial communities that allow expressing and processing distributed computations on massive amounts of data. Multiple scheduling algorithms have been proposed to ensure that short interactive jobs, large batch jobs, and guaranteed-capacity production jobs running on these frameworks can deliver results quickly while maintaining a high throughput. However, only a few works have examined the effectiveness of these algorithms. Objective: The Evidence-based Software Engineering (EBSE) paradigm and its core tool, i.e., the Systematic Literature Review (SLR), have been introduced to the Software Engineering community in 2004 to help researchers systematically and objectively gather and aggregate research evidences about different topics. In this paper, we conduct a SLR of task scheduling algorithms that have been proposed for big data platforms. Method: We analyse the design decisions of different scheduling models proposed in the literature for Hadoop, Spark, Storm, and Mesos over the period between 2005 and 2016. We provide a research taxonomy for succinct classification of these scheduling models. We also compare the algorithms in terms of performance, resources utilization, and failure recovery mechanisms. Results: Our searches identifies 586 studies from journals, conferences and workshops having the highest quality in this field. This SLR reports about different types of scheduling models (dynamic, constrained, and adaptive) and the main motivations behind them (including data locality, workload balancing, resources utilization, and energy efficiency). A discussion of some open issues and future challenges pertaining to improving the current studies is provided.

Enhanced local detouring mechanisms for rapid and lightweight failure recovery in OpenFlow networks

Failure management is one of the important instruments in software-defined networks (SDN) as with traditional networks to ensure uninterrupted and highly reliable services. SDN abstracts the network behaviors with the centralized controller, greatly simplifying network operations and management. So, it is expected from SDN to simplify and reduce the complex and time-consuming failure handling processes. Local detouring is a popular approach for rapid failure handling in traditional networks. Therefore, applying local detouring in SDN has been studied in various literature; however, the approach relies on storing preconfigured alternate paths per ongoing flows and exhausts the SDN controller for the per-flow detouring support. In this paper, we propose enhanced local detouring mechanisms with flow grouping and aggregation methods for the rapid and lightweight failure handling in OpenFlow networks. The proposed mechanisms are evaluated against a controller dependent reactive (CDR) mode, regarding failure response time, SDN controller overhead, and a required number of alternate path rules.

Investigation of Home Agent Load Balancing, Failure Detection and Recovery in IPv6 Network-based Mobility

Mobile IPv6 came as an extensively acknowledged technology to support mobility in networks. Home agents are in charge for the registration of mobile devices and act as a key entity for the tunneling of data packets to the corresponding registered mobile nodes. A single home agent has an administrative control over the critical tasks including home agent registration management, maintenance of cache data and tunneling of data packets to the mobile nodes that are away from their home networks and so on. However in this approach, home agent act as the sole failure point, which gave rise to the placement of multiple home agents to overcome this issue. The load balancing mechanism for multiple home agent deployment faces the problem of improper load sharing, signaling overhead and synchronization issues. Moreover, failure detection and recovery mechanism is inefficient in nature. It experiences a significant delay in tunneling of data packets and suffers from disconnection making it incompetent for the use in real time applications. Hence, this paper investigates and analyzes the various load balancing mechanisms of mobile IPv6. In addition, it presents the comparative study of the failure detection and recovery mechanism of existing methods. Finally, it concludes that future work can be extended in the domain of distributed active load sharing mechanism and proactive failure detection.

Service Failures and Consumer Responses: Exploring the Antecedents of Consumer Dissatisfaction and Propensity to Complain

This qualitative research synthesizes the literature of service failure and consumer complaining behavior and presents a holistic template of service failure and consumer responses. In reviewing the literature, seven propositions are made which include consumer personal factors, causal attributions, situational and organizational factors, service failure recovery perceptions, past relationship, and switching costs that influence the sequential relationship between service failure, consumer dissatisfaction and propensity to complain respectively. A qualitative methodology is employed to ratify the propositions using faceto face structured interviews from thirty individuals having different backgrounds. It was found that the type and intensity of service failure, social stratum, state and type of consumers' emotions, beliefs, personality, attributions, perceptions of the organizational responsiveness, perceived service failure recovery mechanisms, past relationship with the brand/organization and switching costs affect consumer dissatisfaction and propensity to complain. Additional insights are also presented. Overall, this study comprehensively covers the serial responses to service failure, the underlying role of internal and external factors and integrates episodic processes of service failure and consumer complaint behavior in one model.

Recursive SDN for Carrier Networks

Control planes for global carrier networks should be programmable and scalable. Neither traditional control planes nor new SDN-based control planes meet both of these goals. Here we propose a framework for recursive routing computations that combines the best of SDN (programmability through centralized controllers) and traditional networks (scalability through hierarchy) to achieve these two desired properties. Through simulation on graphs of up to 10,000 nodes, we evaluate our design's ability to support a variety of unicast routing and traffic engineering solutions, while incorporating a fast failure recovery mechanism based on network virtualization.

A Virtual Id Routing Protocol for Future Dynamics Networks and Its Implementation Using the SDN Paradigm

In this paper, we propose Virtual Id Routing (VIRO) a novel "plug-&-play" non-IP routing protocol for future dynamics networks. VIRO decouples routing/forwarding from addressing by introducing a topology-aware, structured virtual id layer to encode the locations of switches and devices in the physical topology. It completely eliminates network-wide flooding in both the data and control planes, and thus is highly scalable and robust. VIRO effectively localizes the effect of failures, performs fast re-routing and supports multiple (logical) topologies on top of the same physical network substrate to further enhance network robustness. We have implemented an initial prototype of VIRO using Open vSwitch, and we extend it (both within the user space and the kernel space) to implement VIRO switching functions in VIRO switches. In addition, we use the POX SDN controller to implement VIRO's control and management plane functions. We evaluate our prototype implementation through emulation and in the GENI (the Global Environment for Network Innovations) testbed using many synthetic and real topologies. Our evaluation results show that VIRO has better scalability than link-state based protocols (e.g. OSPF and SEATTLE) in terms of routing-table size and control overhead, as well as better mechanisms for failure recovery.

An Effective Failure Recovery Mechanism for SIP/IMS Services

There are many challenges to provide telecommunication services. The reliability is one of the most important challenges. Telecommunications systems require 99.999% reliability, which means that system only tolerant of downtime about 5 minutes per year. To achieve such high reliability, multiple servers and a shared database are deployed for a Session Initiation Protocol (SIP)/ IP Multimedia Subsystem (IMS) service. This paper investigates the effects of different database (DB) access mechanisms such as Write-Through (WT), Write-Back (WB), and the database-only (DB-only) mechanisms. Then this paper proposes an enhancement for the WB mechanism to improve the successful call setup probability. The analytic models are derived to analyze the performance of the proposed mechanism.

Design and Implementation of Path Failure Detection and Recovery Mechanism for Multihomed HIMALIS Network

We have designed the Heterogeneity Inclusion and Mobility Adaptation through Locator ID Separation (HIMALIS) network architecture to natively support sessions transfer from one link to another independently of the network and transport layer protocols. However, lack of a prompt path failure detection mechanism in HIMALIS cannot well utilize multiple links to protect against path failures. In this paper, we propose a complete path recovery mechanism that consists of path failure detection, lively path exploration, and path recovery processes. In particular, the failure detection mechanism is based on monitoring of data sending and receiving instances in end hosts by employing two timers, probe timer and keepalive timer. In case of path failure caused by gateway's upstream link down, the gateway provides the link failure notification to the host for expediting the detection process. We also present an overview of implementation of the mechanism. The implemented functions are verified by using both TCP and UDP applications and evaluated in a 21-node-scale network-emulation environment. The results show that the proposed mechanism provides fast failure detection, about 7 s which is much faster than the failure detection carried out based on TCP retransmission timeout which may take 5 min in the worst case, under both site- and host-multihoming configurations. The results also show that the proposed mechanism takes only about 200 ms to detect gateway's upstream link down.

Service Failure Recovery and Prevention: Managing Stockouts in Distribution Channels

In managing service failures such as stockouts, most research has emphasized preventive mechanisms, whereas stockout recovery mechanisms have been largely ignored. We propose and examine a failure-recovery mechanism (i.e., contractual stockout recovery) in the presence of demand uncertainty and compare it with failure-prevention mechanisms in a dyadic distribution channel. We find that stockout recovery mechanisms can improve channel profitability under certain conditions. More importantly, we find that stockout recovery may outperform stockout prevention mechanisms such as return policy and vendor managed inventory in improving manufacturer and channel profitability. This is because stockout recovery reduces channel-wide stockout risks and allows benefits from the reduced risks to be shared between the manufacturer and the retailer, helping alleviate double marginalization. Although return policy also reduces stockout risks, it does so by increasing inventory risks in the channel without reducing channel exposure to demand uncertainty. Thus, our research suggests that stockout recovery can be an effective alternative in managing stockouts to those common methods of stockout prevention mechanisms.

Instability of sharing systems in the presence of retransmissions

Retransmissions represent a primary failure recovery mechanism on all layers of communication network architecture. Similarly, fair sharing, for example, processor sharing (PS), is a widely accepted approach to resource allocation among multiple users. Recent work has shown that retransmissions in failure-prone, for example, wireless ad hoc, networks can cause heavy tails and long delays. In this paper, we discover a new phenomenon showing that PS-based scheduling induces complete instability with zero throughput in the presence of retransmissions, regardless of how low the traffic load may be. This phenomenon occurs even when the job sizes are bounded/fragmented, for example, deterministic. Our analytical results are further validated via simulation experiments. Moreover, our work demonstrates that scheduling one job at a time, such as first-come-first-serve, achieves a larger stability region and should be preferred in these systems.

Fast reroute from single link and single node failures for IP multicast

The rise in multicast implementations has seen with it an increased support for fast failure recovery from link and node failures. Most recovery mechanisms augment additional services to existing protocols causing excessive overhead, and these modifications are predominantly protocol-specific. In this paper, we develop a multicast failure recovery mechanism that constructs protocol independent fast reroute paths to recover from single link and single node failures. We observe that single link failure recovery in multicast networks is similar to recovering unicast traffic, and we use existing unicast recovery mechanisms for multicast traffic. We construct multicast protection trees that provide instantaneous failure recovery from single node failures. For a given node x, the multicast protection tree spans all its neighbors and does not include itself. Thus, when the node fails, the neighbors of the node are connected through the multicast protection tree instead of node x, and forward the traffic over the multicast protection tree for the duration of failure recovery. The multicast protection trees are constructed a priori, without the knowledge of the multicast traffic in the network. Based on simulations on three realistic network topologies, we observe that the multicast protection trees increase the routing table size only by 38% on average and the path length between any source–destination pair by 13% on average.

A UTP semantic model for Orc language with execution status and fault handling

The Orc language is a concurrency calculus proposed to study the orchestration patterns in service oriented computing. Its special features, such as high concurrency and asynchronism make it a brilliant subject for studying web applications that rely on web services. The conventional semantics for Orc does not contain the execution status of services so that a program cannot determine whether a service has terminated normally or halted with a failure after it published some results. It means that this kind of failure cannot be captured by the fault handler. Furthermore, such a semantic model cannot establish an order saying that a program is better if it fails less often. This paper employs UTP methods to propose a denotational semantic model for Orc that contains execution status information. A failure handling semantics is defined to recover a failure execution back to normal. A refinement order is defined to compare two systems based on their execution failures. Based on this order, a system that introduces a failure recovery mechanism is considered better than one without. An extended operational semantics is also proposed and proven to be equivalent to the denotational semantics.

Power consumption reduction through elastic data rate adaptation in survivable multi-layer optical networks

Network survivability requires the provisioning of backup resources in order to protect active traffic against any failure scenario. Backup resources, however, can remain unused most of the time while the network is not in failure condition, inducing high power consumption wastage, if fully powered on. In this paper, we highlight the power consumption wastage of the additional resources for survivability in IP/multi-protocol label switching (MPLS) over dense wavelength division multiplexing multi-layer optical networks. We assume MPLS protection switching as the failure recovery mechanism in the network, a solution interesting for current network operators to ensure fast recovery as well as fine-grained recovery treatment per label switched path. Next, we quantitatively show how elastic optical technologies can effectively reduce such a power consumption by dynamically adjusting the data rate of the transponders to the carried amount of traffic.

SERVER FAILURES ENABLED JAVASPACES SERVICE

JavaSpaces service is a Distributed Shared Memory (DSM) implementation. It has been introduced by Sun Microsystems as a service of the Jini system. Currently, JavaSpaces support client side fault tolerance. It enables both transaction and mobile coordination mechanisms for such purpose. The application failures could be detected and recovered. However, server side failures may occur during the application runtime. Therefore, it is important to supply JavaSpaces with a mechanism that handles such type of failures dynamically. On the other hand, An example of a system that supports both server and client fault tolerance over DSM is TRIPS system. TRIPS protocols are suitable to be integrated in JavaSpaces to supply it with server fault tolerance capabilities. In this study, a server Failures Enabled Javaspaces Service (FTJS) is introduced. FTJS is based on the dynamic failure detection and recovery mechanisms implemented by TRIPS. However, FTJS is able to handle both client and server side failures. The analysis, design and implementation issues of FTJS are introduced.

Realizing a fault-tolerant embedded controller on distributed real-time systems

Advances in real-time, embedded and distributed systems along with control and communication theory have catalyzed the rapid emergence of cyber-physical systems such as a self-driving car. The importance of fault-tolerance support on a cyber-physical system (CPS) has been greatly emphasized by recent research due to the nature of CPS that senses its surroundings, processes sensor data, and reacts using its actuators. In order to tackle this challenge, we proposed SAFER (System-level Architecture for Failure Evasion in Real-time Applications) in our previous work. SAFER is able to tolerate fail-stop processor and/or task failures for distributed embedded real-time systems. One of its limitations, however, is that SAFER is not capable of tolerating a failure of a processor with a dedicated connection to an actuator. This paper provides a method that relaxes this limitation by (1) deploying a small piece of hardware to avoid a dedicated connection between a processor and an actuator, (2) adding a software module that monitors and controls the hardware, and (3) enhancing the failure detection and recovery mechanisms of SAFER to support these changes. The detailed implementation and evaluation of the SAFER extension is on-going work.

Failure recovery in distributed model composition with intelligent assistance

Composite models in Decision Support Systems (DSS) are combinations of model functions to solve complex decision problems. They must be executed successfully to obtain the desired results. Unfortunately, faults may happen during its execution due to the dynamic networks or the information asymmetry between developers and users. Therefore, designing effective failure recovery mechanism to ensure the reliability of the composite model execution is essential. Progress has been made in the web service composition field to obtain limited failure recovery capabilities, but not fully applicable to composite models since model services are informational services in essence which do not change the world conditions. This paper proposes a unified framework, integrated with a process ontology and multiple recovery strategies that can provide valuable failure recovery recommendations intelligently. The recovery strategies have been greatly enhanced in providing greater failure recovery capability. Feasibility and efficiency of this framework have been illustrated and tested.

SSR-CDPS: A Novel Continuous Data Protection System Supporting Service Recovery

With the rapid development of computer networks, invasion and attack becomes more and more frequent in network environment. Therefore, insuring the information system continuously and correctly working has become an urgent problem in the information security field. And continuous data protection (CDP) supporting service recovery provides us a good idea to resolve it. In this article, we have designed and implemented a continuous data protection system supporting service recovery, which is named SSR-CDPS. The basic design idea of the SSR-CDPS architecture is dividing the whole protection process into three stages: monitoring, protecting and recovery. Meanwhile, we propose a multi versioning data backup mechanism for the protecting process and a novel file failure recovery mechanism. Especially, we transfer the full copy operation of file contents into the merely slight modification operation on the FAT table, MFT table or other similar data structures; so that we achieve the aim of greatly reducing the direct disks read and write overhead. The test toward the SSR-CDPS prototype system shows that the system can significantly improve the efficiency of data protection within limited change to the existing file system; thus, it can meet the high real-time and the continuity requirements of the information system recovery applications.