Failure Recovery Research Articles

A proactive controller failure recovery mechanism in SD-WAN with multiple controllers

The separation of data and control plane is a key feature of Softwaredefined networking (SDN), which makes network administration more intelligent. The control plane is realized using a logically centralized controller. In SD-WAN, as the network becomes larger, multiple controllers are needed to manage the network. In this scenario, there is a chance for controller failure due to overload. Once the controller fails, the switches lose connection with the controller. The load of the failed controller is to be re-distributed among other controllers. Sometimes this load transfer may cause many other problems like controller chain failure. It may consistently undermine the trustworthiness of the network. In the proposed technique, the multiple controller deployment based on affinity propagation clustering in SD-WAN is extended to include a proactive controller failure recovery mechanism. Whenever the controller load exceeds a pre-defined threshold, a set of switches under the bottleneck controller will be reassigned to a neighbouring controller without exceeding its capacity. The simulation results show that when network traffic increased, the proposed proactive controller failure recovery method balanced the controller load, resolved a cascading controller failure, improved the average throughput, and reduced the average end-to-end delay and packet loss effectively.

Cascading failure resilience analysis and recovery of automotive manufacturing supply chain networks considering enterprise roles

With the rapid development of global trade, the member enterprises in the automotive manufacturing supply chain are widely spread around the world. As a result, the scale of the automotive manufacturing supply chain network has become increasingly large, and the structure has become more complex. At the same time, the connection between enterprises is becoming closer. On one hand, the close business relationships between enterprises and the vast network structure accelerate the operation efficiency of the automotive manufacturing supply chain network. On the other hand, it also exposes the entire network system to high risks of cascading failures when facing disruptions due to enterprise production interruptions. Therefore, this work aims to establish a realistic cascading failure model for the automotive manufacturing supply chain network to explore its cascading failure characteristics. Based on this model, a two-stage recovery strategy is proposed to enhance the network’s cascading failure resilience. Experimental results demonstrate the following findings: (1) underload nodes that emerge after load redistribution are the main cause of cascading failure resilience degradation in automotive manufacturing supply chain networks; (2) cascading failure resilience is negatively correlated with the load-exponential coefficient, the capacity lower-limit coefficient, and the attenuation-tolerance coefficient; (3) among four different types of attacks, cascading failures caused by the high eigenvector centrality attack strategy and the high degree attack strategy impose the most severe damage on the automotive manufacturing supply chain network; (4) the proposed two-stage recovery strategy can significantly improve the cascading failure resilience of the automotive manufacturing supply chain network.

Quantitative measurement of link failure reaction time for devices with P4-programmable data planes

Quick handling of link failures remains a challenging issue in current communication networks, although it is crucial to many routing algorithms. Link failures are the leading cause of packet losses and delays, therefore, failure recovery is tied to stringent requirements for certain services, such as the sub-50 millisecond completion time for carrier-grade networks, which is sometimes difficult to achieve in traditional routing schemes. For this reason, fast recovery strategies are key pillars of modern communication networks. In this paper, we demonstrate the benefits of the devices with Programmable Data Planes (PDP) for fast reacting to link failures. We first review the link failure detection, reaction and recovery procedures and then we discuss the main fast failure recovery mechanisms employed by different types of devices in current communication networks. In addition, we present a novel method to measure the link failure reaction time of an Intel Tofino switch with PDP, as well as the results obtained when measuring such time using real hardware equipment. Our results show that such hardware devices provide a failure reaction time in the order of microseconds, with an average of 472.88μ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\upmu $$\\end{document}s, which poses PDP as a key technology to achieve zero packet loss and zero delay failure recovery.

Resilience improvement of cyber-physical supply chain networks considering cascading failures with mixed failure modes

Failure mode analysis and recovery strategy improvement are critical to maintain the resilience of cyber-physical supply chain networks (CPSCNs). However, current literature mostly focuses on single failure mode and single-stage recovery strategy for CPSCNs, which may not be the case for real-world CPSCNs. In this paper, a new cascading failure model with four mixed failure modes and a joint improvement model for two-stage recovery regarding CPSCNs are proposed. A CPSCN model consisting of a weighted undirected cyber-supply network and a weighted directed physical-supply network is established to investigate the mixed cascading failures. Given the recovery requirements of practical contracts, the recovery time constraint on the enterprises in CPSCNs is considered in the joint improvement model of two-stage recovery. Based on the case studies of synthetic networks and a European supply chain network, the effects of some key factors, such as network topology and node coupling modes, on the resilience of CPSCNs are discussed. The performance of the proposed recovery strategy is compared with two traditional network recovery strategies, and the results show that the proposed strategy can help restore network performance more effectively. This work provides insights for CPSCNs recovery with resilience improvement against future cascading failures.

Joint Reconfiguration after Failure for Performing Emblematic Gestures in Humanoid Receptionist Robot.

This study proposed a strategy for a quick fault recovery response when an actuator failure problem occurred while a humanoid robot with 7-DOF anthropomorphic arms was performing a task with upper body motion. The objective of this study was to develop an algorithm for joint reconfiguration of the receptionist robot called Namo so that the robot can still perform a set of emblematic gestures if an actuator fails or is damaged. We proposed a gesture similarity measurement to be used as an objective function and used bio-inspired artificial intelligence methods, including a genetic algorithm, a bacteria foraging optimization algorithm, and an artificial bee colony, to determine good solutions for joint reconfiguration. When an actuator fails, the failed joint will be locked at the average angle calculated from all emblematic gestures. We used grid search to determine suitable parameter sets for each method before making a comparison of their performance. The results showed that bio-inspired artificial intelligence methods could successfully suggest reconfigured gestures after joint motor failure within 1 s. After 100 repetitions, BFOA and ABC returned the best-reconfigured gestures; there was no statistical difference. However, ABC yielded more reliable reconfigured gestures; there was significantly less interquartile range among the results than BFOA. The joint reconfiguration method was demonstrated for all possible joint failure conditions. The results showed that the proposed method could determine good reconfigured gestures under given time constraints; hence, it could be used for joint failure recovery in real applications.

Service mega-disruptions: a conceptual model and research agenda

PurposeThis paper aims to formally conceptualize service mega-disruptions as any far-reaching and unforeseen general environmental stressor or threat that impacts a service organization’s ability to provide a desired level of service. The authors differentiate sudden large-scale general environmental threats from traditional service failures in scope and scale of impact via number of customers and sectors affected and duration and speed of the disruption.Design/methodology/approachThis paper draws from service recovery theory to build a conceptual model of service mega-disruptions. The resulting conceptual model maps service failure recovery strategies against a service mega-disruption recovery approach to examine consumer response to changes in service value. This work further articulates additional research needs including conceptualization, measurement and methods as traditional drivers of service recovery and the value of the service experience change in response to service mega-disruptions.FindingsThis work proposes a research agenda to investigate whether service mega-disruptions can bypass the need for service recovery due to a consumer self-moderating process. As past research shows, the less control a service provider has over a failure, the more customers attribute fault to the situation and transfer blame away from an organization. This paper suggests that this self-moderating process disrupts the need for service providers to court forgiveness for a failure with perceptions of similarity and controllability providing an alternate pathway to customer forgiveness. Similarly, it is suggested that service mega-disruptions play a role in transforming service ecosystems into tighter, more contractual systems with less agency for service providers and poorer ability to adjust to market conditions. The duration and longevity of effects on service providers’ control, agency and ability to adjust following a service mega-disruption must be researched further.Originality/valueThis paper builds theory to develop a conceptual model of service mega-disruptions and their role in customer engagement and reshaping the service ecosystem. This paper culminates in the proposition of a research agenda that aims to build research capacity among services marketing scholars as service providers’ coordination and market conditions are challenged by service mega-disruptions.

Abstract 17254: Myocardial Functional Recovery Following Durable Ventricular Assist Device in Children

Introduction: Ventricular Assist Device (VAD) explantation following myocardial functional recovery (MFR) for heart failure (HF) is uncommon and associated with a risk of recurrence of HF. Research Question: What are the patient characteristics and outcomes for VAD explantation in children following MFR? Methods: Retrospective, single centre study of patients who were supported with durable VADs, both intracorporeal continuous flow devices (CFD) and paracorporeal pulsatile flow devices (PFD) between 2004 to 2022. Results: A total of 75 children, of which 43 were female, underwent VAD implantation (PFD = 61 and CFD = 14) at a median (IQR) age of 5.6 (0.8, 13.5) years and with a weight of 16.2 (7.5, 40.7) kg. From this cohort, we identified 8/75 (11%) children who underwent VAD explantation for MFR after a median duration of 56 (22, 115) days. All were supported with durable PFD. Five were listed for transplantation as a part of their HF management strategy following VAD implantation and four patients had dilated cardiomyopathy (DCM). Of those, 7/8 (88%) children with MFR were under 2 years of age and 5/8 (63%) were supported for > 90 days. The pathological findings on the LV core samples were variable in severity and no patient had any evidence of active inflammation (see Table). After explant, 7/8 patients remained in HF remission with no symptoms and stable LV function. One patient had a recurrence of HF following explantation after demonstrating MFR while on VAD support. This patient was 5 years old at implant and was in HF remission for ~9 months following explant before HF recurrence, requiring VAD re-implantation and eventual heart transplantation. Conclusions: MFR resulting in VAD explantation is feasible in children with chronic HF especially for those < 2 years of age. Further work is needed to help better identify the features that promote MFR and maintain it after explant.

Abstract 15777: Vascular Density and Fibrosis in Heart Failure Recovery

Introduction: Chronic unloading with a left ventricular assist device (LVAD) leads to LV remodeling including increased endothelial density, reduction of fibrosis, and improved cardiac function. Specific mechanisms of such recovery have not been fully explored in humans. Hypothesis: We hypothesize that recovery from non-ischemic heart failure is accompanied by increased vascular density and reduction in fibrosis. Methods: We developed and characterized a murine model of non-ischemic heart failure (HF) and recovery (HFR) using Angiotensin II infusion pumps and L-NAME/NaCl in water for 5 weeks followed by removal of these insults to induce a recovery. Ejection fraction, LV mass, fibrosis, and vessel density were assessed at various time points. We quantified vessel density using fluorescent microsphere bead perfusion and measured fibrosis using Masson’s Trichrome staining. Statistical analysis was performed using One way ANOVA, n=5 mice/group with 2 replicate experiments. ( p <0.05 was considered significant.)Results: LV mass and interstitial fibrosis increased significantly while ejection fraction and vascular density showed significant decreases in HF vs control mice while recovering hearts showed normalized ejection fraction and LV mass with a significant reduction in fibrosis. There was a very significant increase in vascular density when compared to HF mice (Fig. 1). Together, these preliminary data support our hypothesis that recovery from heart failure involves a vascular recovery. Conclusions: Morphological recovery of cardiac function is accompanied by an increase in vascular density along with a reduction in fibrosis in a non-ischemic mouse model of heart failure recovery. This increase in vascular density may be a required component of the recovery process from heart failure and further studies are necessary to understand the specifics of such mechanisms.

Seeking empathy or suggesting a solution? Effects of chatbot messages on service failure recovery

Chatbots as prominent form of conversational agents are increasingly implemented as a user interface for digital customer-firm interactions on digital platforms and electronic markets, but they often fail to deliver suitable responses to user requests. In turn, individuals are left dissatisfied and turn away from chatbots, which harms successful chatbot implementation and ultimately firm’s service performance. Based on the stereotype content model, this paper explores the impact of two universally usable failure recovery messages as a strategy to preserve users’ post-recovery satisfaction and chatbot re-use intentions. Results of three experiments show that chatbot recovery messages have a positive effect on recovery responses, mediated by different elicited social cognitions. In particular, a solution-oriented message elicits stronger competence evaluations, whereas an empathy-seeking message leads to stronger warmth evaluations. The preference for one of these message types over the other depends on failure attribution and failure frequency. This study provides meaningful insights for chatbot technology developers and marketers seeking to understand and improve customer experience with digital conversational agents in a cost-effective way.

Therapeutic potential of human induced pluripotent stem cell–derived cardiac tissue in an ischemic model with unloaded condition mimicking left ventricular assist device

ObjectiveThis study aimed to explore the therapeutic potential of human induced pluripotent stem cell (hiPSC)-derived cardiac tissues (HiCTs) in the emerging approach of bridge to recovery for severe heart failure with ventricular assist devices. We used a rat model of heterotopic heart transplantation (HTx) to mimic ventricular assist device support and heart unloading. MethodsHiCTs were created by inserting gelatin hydrogel microspheres between cell sheets made from hiPSC-derived cardiovascular cells. Male athymic nude rats underwent myocardial infarction (MI) and were divided into the following groups: MI (loaded, untreated control), MI + HTx (unloaded, untreated control), MI + HTx + HiCT (unloaded, treated), and MI + HiCT (loaded, treated). HiCTs were placed on the epicardium of the heart in treated groups. We evaluated HiCT engraftment, fibrosis, and neovascularization using histologic analysis. ResultsAfter 4 weeks, HiCTs successfully engrafted in 5 of 6 rats in the MI + HTx + HiCT group (83.3%). The engrafted HiCT area was greater under unloaded conditions (MI + HTx + HiCT) than loaded conditions (MI + HiCT) (P < .05). MI + HTx + HiCT had a significantly smaller infarct area compared with MI and MI + HTx. The MI + HTx + MiCT group exhibited greater vascular density in the border zone than MI and MI + HTx. HiCT treatment suppressed cardiomyocyte atrophy due to left ventricular unloading (P = .001). The protein level of muscle-specific RING finger 1, an atrophy-related ubiquitin ligase, was lower in the MI + HTx + HiCT group than in MI + HTx (P = .036). ConclusionsTransplanting HiCTs into ischemic hearts under unloaded conditions promoted engraftment, neovascularization, attenuated infarct remodeling, and suppressed myocyte atrophy. These results suggest that HiCT treatment could contribute to future advancements in bridge to recovery.

Heart Failure Recovery After Lumasiran and Isolated Kidney Transplantation in Primary Hyperoxaluria Type 1

Heart Failure Recovery After Lumasiran and Isolated Kidney Transplantation in Primary Hyperoxaluria Type 1

Ensuring reliable network operations and maintenance: The role of PMRF for switch maintenance and upgrades in SDN

In the realm of software-defined networking (SDN), ensuring robust maintenance operations and the consistent performance of switch operations, especially during maintenance and upgrades, is of paramount importance. Traditional SDN methods, both reactive and proactive, face challenges like extended convergence times, inefficient use of Ternary Content-Addressable Memory (TCAM) space, and unpredictable maintenance time. To tackle these issues and enhance maintenance operations, this paper introduces the Preventive Maintenance Routing Framework (PMRF) – a novel approach designed for uninterrupted upgrades in SDN settings. Our framework focuses on selecting reliable backup paths and timely isolation of the maintenance switch except critical switch, framed as an NP-hard mixed-integer linear programming problem. To address this complexity, we present a multi-objective heuristic algorithm that efficiently determines the best path for upcoming affected flows. Testing shows that, in varied network conditions, PMRF markedly reduced packet drops, potentially enhancing SDN operation efficiency, minimizing end-to-end delays, and maintaining failure recovery times below 50 ms and efficiently manage TCAM resources. In summary, PMRF offers a robust solution for optimizing network reliability and efficiency during maintenance operations and upgrades in core networks.

Research Status and Thinking of Service Failure and Service Recovery of Service Robot

With the development of intelligent technology, service robots have appeared in various service first-line environments, and consumers will inevitably encounter service failure of service robots. This paper analyzes the types of service failure by combing the relevant literature on service failure and service recovery, and based on the social cognitive theory, clarifies the remedial measures that should be taken after service robot fails. It is found that after process failure, the service robot makes emotional response such as apology or empathy, the service recovery effect is better; When the result fails, the service robot makes rational response such as explanation, the consumer's attitude is more positive. The research results provide practical enlightenment for enterprises to carry out service recovery.

A faster failure recovery mechanism for Multipath TCP

A faster failure recovery mechanism for Multipath TCP

Reliability-aware failure recovery for cloud computing based automatic train supervision systems in urban rail transit using deep reinforcement learning

As urban rail transit construction advances with information technology, modernization, information, and intelligence have become the direction of development. A growing number of cloud platforms are being developed for transit in urban areas. However, the increasing scale of urban rail cloud platforms, coupled with the deployment of urban rail safety applications on the cloud platform, present a huge challenge to cloud reliability.One of the key components of urban rail transit cloud platforms is Automatic Train Supervision (ATS). The failure of the ATS cloud service would result in less punctual trains and decreased traffic efficiency, making it essential to research fault tolerance methods based on cloud computing to improve the reliability of ATS cloud services. This paper proposes a proactive, reliability-aware failure recovery method for ATS cloud services based on reinforcement learning. We formulate the problem of penalty error decision and resource-efficient optimization using the advanced actor-critic (A2C) algorithm. To maintain the freshness of the information, we use Age of Information (AoI) to train the agent, and construct the agent using Long Short-Term Memory (LSTM) to improve its sensitivity to fault events. Simulation results demonstrate that our proposed approach, LSTM-A2C, can effectively identify and correct faults in ATS cloud services, improving service reliability.

Comparison of immediate withdrawal and stepwise reduction in duration of non-invasive ventilation in chronic obstructive pulmonary disease patients presenting with acute hypercapnic respiratory failure.

There is no universally acceptable protocol for the withdrawal of non-invasive ventilation (NIV) in chronic obstructive pulmonary disease (COPD) patients presenting with acute hypercapnic respiratory failure (AHcRF). This study was carried out to evaluate immediate against stepwise reduction in NIV. Sixty COPD patients with AHcRF who were managed with NIV were randomized into two groups - immediate NIV withdrawal (Group A), and stepwise reduction of NIV duration (Group B). The rate of successful NIV withdrawal, time to recurrence of hypercapnic respiratory failure, total duration of NIV use, and hospital length of stay (LOS), were compared among the 2 groups. NIV was successfully withdrawn in 51/60 (85%) patients. NIV was successfully withdrawn in 24/30 (80%) patients in Group A and 27/30 (90%) patients in Group B (p=0.472). The total duration of NIV use was significantly lower in Group A (38.97±17 hours) as compared to Group B (64.3±7.74 hours) (p<0.0001). The hospital LOS was significantly lower in group A (5.8±1.6 days) as compared to Group B (7.7±0.61 days) (p<0.0001). To conclude, immediate withdrawal of the NIV after recovery of respiratory failure among patients with exacerbation of COPD is feasible and does not increase the risk of weaning failure.

On Locality of Some Binary LCD Codes

The design of codes for distributed storage systems that protects from node failures has been studied for years, and locally repairable code (LRC) is such a method that gives a solution for fast recovery of node failures. Linear complementary dual code (LCD code) is useful for preventing malicious attacks, which helps to secure the system. In this paper, we combine LRC and LCD code by integration of enhancing security and repair efficiency, and propose some techniques for constructing LCD codes with their localities determined. On the basis of these methods and inheriting previous achievements of optimal LCD codes, we give optimal or near-optimal [n, k, d, r] LCD codes for k ≤ 6 and n ≥ k+1 with relatively small locality, mostly r ≤ 3. Since all of our obtained codes are distance-optimal, in addition, we show that the majority of them are r-optimal and the other 63 codes are all near r-optimal, according to CM bound.

Dynamic Reliability Assessment Method for a Pantograph System Based on a Multistate T-S Fault Tree, Dynamic Bayesian

The operational reliability of rail vehicle pantograph systems is evaluated by transforming T-S multistate fault trees into dynamic Bayesian networks (DBNs), which take into account system multistability, long-lasting operation, dynamic failure, and maintenance recovery. The T-S multistate fault tree structure is constructed by the content validity ratio and content validity index; the T-S gate rule expressing causal uncertainty is constructed by using fuzzy theory and dependent uncertain ordered weighted averaging expert scoring, and finally, the pantograph T-S multistate fault tree is transformed into a DBN model characterizing the dynamic interaction and time dependence of the system. The dynamic evolution laws of reliability of a pantograph system in maintenance and maintenance-free states over time are inferred, compared and analyzed. The results show that the system availability of a pantograph system decreases continuously during 720 days of operation. The system availability without maintenance decreases to 0.881, and the system availability with maintenance is 0.952. The reliability of a pantograph system can be effectively ensured with maintenance during the operation period; the sensitivity analysis is performed by changing the failure rate of the equipment to 120% or 80%; the fall indicator, the electrical control box, and the elevating bow motor are the weak links in the system, and the impact of fault escalation on the reliability of a pantograph system is analyzed. It is then verified that the system reliability can be further improved by using a preventive maintenance strategy, and the steady-state reliability can be gradually reached, which is about 0.9968, providing a reference for the maintenance of a pantograph system.

An Influence Model of the Human-Automation Team: Effects of Workload and Automation Reliability, Transparency and Degree

Four important variables that influence performance of a human-automation team (HAT) are the workload (resource) demands of the task and environment, and the reliability, degree, and transparency of the automation that assists the human. In this paper, first, we present a model that predicts how these variables influence the human contribution to performance of the HAT, as mediated by changes in trust, dependence, and situation awareness (SA) and experienced workload (dependent variables). These factors vary in their strength of influence and their interaction with each other. Then we describe in greater depth how a meta- analysis of 50 studies has revealed differences in the strength of influence of automation transparency on those dependent variables. In particular, transparency has a large effect on improving the accuracy of performance, increasing trust and situation awareness. Transparency has different benefits for performance in routine situations, where it improves accuracy of the human-automation team, than for situations when automation unexpectedly fails, where it decreases the time for failure recovery. Finally, we illustrate how the model accommodates differences in discrimination task difficulty revealed in an experiment on decision aiding for nautical collision avoidance. This shows the benefits of the model in predicting the tradeoff of factors influencing human-automation team performance.

An improved prediction method of subsequent commutation failure of an LCC-HVDC considering sequential control response

Subsequent commutation failure (SCF) can be easily generated during the first commutation failure (CF) recovery process in a line-commutated converter-based high voltage direct-current system. SCF poses a significant threat to the safe and stable operation of power systems, and accurate prediction of CF is thus important. However, SCF is affected by the operating characteristics of the main circuit and the coupling effects of sequential control response in the inverter station. These are difficult to predict accurately. In this paper, a new SCF prediction method considering the control response is proposed based on the physical principle of SCF. The time sequence and switching conditions of the controllers at different stages of the first CF recovery process are described, and the corresponding equations of commutation voltage affected by different controllers are derived. The calculation method of the SCF threshold voltage is proposed, and the prediction method is established. Simulations show that the proposed method can predict SCF accurately and provide useful tools to suppress SCF.