Failure Detector Research Articles

Quantifying mobile robot localization safety for an EKF-based SLAM estimator: An integrity monitoring approach

The safety of SLAM-based estimation for mobile robots is a challenging research problem, particularly for life- or mission-critical exploratory applications. To address this problem, this work proposes a method to utilize integrity risk, a widely used performance metric in aviation, to quantify SLAM-based mobile robot’s localization safety. More importantly, the approach accounts for sensor measurement faults, unknown deterministic errors that cannot be modeled via Gaussian white noise. The method is tailored for an EKF-based SLAM estimator, a chi-squared failure detector, and a local nearest neighbors data association criterion. The results show that data association errors can cause significant positioning performance degradation that can only be predicted using the proposed integrity risk metric. Furthermore, the study demonstrates that as the map’s landmark density increases, mobile robot localization safety improves, except when the landmark map’s density is too high to make the features indistinguishable, which leads to positioning safety degradation.

Investigation on the failure of the space HgCdTe detector with long-term storage tests

The HgCdTe detector is the core device of the infrared imaging system, its reliability directly restricts the performance of the whole imaging system. For some special applications, it maintains in a non-operating storage state for a long time, it is significant to study its storage reliability. Due to the long lifetime of detectors, most studies have been conducted with accelerated life tests, there is a lack of experimental information concerning the failure of these detectors after longtime storage. In this article, the performance degradation of Hg1-xCdxTe detector with the 127 months long-term storage test is studied. The failure mode and failure mechanism of detectors are investigated through the statistics analysis and optical photomicrography. The results indicate that there are three failure modes, such as low-temperature opening circuit, chip circuit opening, and signal reduction accompanied by increased noise. The main type of failure is in the mode of signal reduction accompanied by increased noise, which is a typical failure mode for long-term storage of HgCdTe detectors. In this typical failure mode, corrosion products are observed near the photosensitive surface in the failure detector, which prove to be a metal product of indium. The long-term storage results can provide data support for accelerating storage test and improve the HgCdTe detectors design.

Proactive Drive Failure Prediction for Cloud Storage System Through Semi-Supervised Learning

Proactive drive failure prediction can help operators handle the failing drives in advance, enhancing the storage system dependability. SSD and HDD failure prediction techniques are currently evolving towards a semi-supervised approach. In this paper, we are dedicated to enhancing the methodology for semi-supervised drive failure prediction from these aspects: design more powerful, robust yet generic models, mine drive failure modes and make the prediction model interpretable. Specifically, we propose two semi-supervised drive failure prediction models, DFP-VL and DFP-FL, from the perspective of reconstructing SMART data and learning SMART data's probability density, respectively. They capture the pattern of healthy drives, and failing drives can be detected when they deviate from the normal pattern. We mine two failure modes, slow deterioration and dramatic deterioration. Based on that, we design two failure detectors, ”ThresholdDetector” and ”HybridDetector”, to determine whether a drive deviates from the normal pattern. We evaluate the proposed methods on both SSD and HDD SMART data. DFP-VL is generic yet effective and can interpret the predicted results. DFP-FL has better performance than DFP-VL, but it cannot interpret the results. ThresholdDetector has low complexity and can detect most failing drives. HybridDetector has high complexity and can further improve the detection performance.

Evaluation of the Reliability of a Standby Redundancy System Under Real Conditions

This paper presents a novel analysis of a standby redundancy system under real conditions. The study considers the presence of a real commutator and failure detector in the system. Through a comprehensive failure mode analysis, mathematical relationships between different module characteristics are established. The results of investigation provide valuable insights for manufacturers, allowing them to evaluate the Mean Time To Failure (MTTF) of the system during the design phase and make informed decisions regarding the selection of failure rates for the detector and commutator. Overall, this work contributes to the effective operation of standby redundancy systems in practical applications.

An in-depth and insightful exploration of failure detection in distributed systems

In today’s world, everyone wants a good profit with a tiny investment and distributed computing is a boon for this purpose. Cloud computing, fog computing, and the Internet of Things (IoT) are well-known examples of distributed computing which provide good computing services and performance. However, providing reliable services in a real environment, which is failure-prone, remains a challenge. To address this issue, failure detectors are used in distributed systems, which are abstract modules responsible for detecting and monitoring the activity of nodes in order to determine whether they are faulty or not. In this paper, an approach is presented for the systematic literature review of failure detectors in distributed systems. Further, many existing review and survey papers on failure detectors are critically analyzed along with their key contributions and limitations. The classification of distributed systems is presented on the basis of the nodes’ properties and the components of system models are described in detail. Various issues and challenges related to agreement and failure problems are also explored. The strengths and limitations of various existing failure detectors are discussed along with their comparative evaluation. Finally, fault-tolerance and recovery techniques are discussed and analyzed.

Self-stabilizing indulgent zero-degrading binary consensus

Guerraoui proposed an indulgent solution for the binary consensus problem. Namely, he showed that an arbitrary behavior of the failure detector never violates safety requirements even if it compromises liveness. Consensus implementations are often used in a repeated manner. Dutta and Guerraoui proposed a zero-degrading solution, i.e., during system runs in which the failure detector behaves perfectly, a node failure during one consensus instance has no impact on the performance of future instances.Our study, which focuses on indulgent zero-degrading binary consensus, aims at the design of an even more robust communication abstraction. We do so through the lenses of self-stabilization—a very strong notion of fault-tolerance. In addition to node and communication failures, self-stabilizing algorithms can recover after the occurrence of arbitrary transient faults; these faults represent any violation of the assumptions according to which the system was designed to operate (as long as the algorithm code stays intact).This work proposes the first, to the best of our knowledge, self-stabilizing algorithm for indulgent zero-degrading binary consensus for time-free message-passing systems prone to detectable process failures. The proposed algorithm recovers within a finite time after the occurrence of the last arbitrary transient fault. Since the proposed solution uses an Ω failure detector, we also present the first, to the best of our knowledge, self-stabilizing asynchronous Ω failure detector, which is a variation on the one by Mostéfaoui, Mourgaya, and Raynal.

Stab-FD: A cooperative and adaptive failure detector for wide area networks

Failure detectors (FDs) are a fundamental abstraction that plays a central role in the design of distributed systems. FDs are distributed oracles that provide processes with unreliable information about process failures, often in the form of a list of trusted or suspected process identities. In this article, we propose a timer-based FD which assesses the quality of its input links, and exchanges its local estimations with other nodes. Nodes use this information to adjust their timers dynamically. Capturing the variations in the quality of each link reduces the number of false suspicions without degrading failure detection time. We present experiments on a dataset of real traces collected on PlanetLab, and compare our approach to well-known state-of-the-art algorithms. Our results show that our new algorithms yield a good trade-off in terms of failure detection speed and accuracy in real scenarios.

Scalable atomic broadcast: A leaderless hierarchical algorithm

Atomic Broadcast is an essential broadcast primitive as it ensures the consistency of distributed replicas. However, it is notoriously non-scalable. In this work, we introduce the Leaderless Hierarchical Atomic Broadcast (LHABcast) algorithm, which has two properties to improve scalability. First, it is a fully decentralized algorithm that does not rely on a sequencer/leader, which is often a significant bottleneck. Processes running LHABcast send messages with local sequence numbers and order messages received from other processes using timestamps inspired on Lamport's logical clocks. A process that receives the required set of timestamps can make a decision about the overall sequence of message delivery. Second, the algorithm is hierarchical: processes are organized on a vCube logical overlay network, which has several logarithmic properties and allows the construction of autonomous spanning trees. vCube also works as a failure detector, assuming crash faults and an asynchronous system model. In this paper, LHABcast is described, specified, and proven to be correct. Both simulation and experimental results are presented. A comparison with an all-to-all strategy shows that the number of messages sent by LHABcast is significantly lower in both fault-free and faulty scenarios. An implementation of LHABcast in Akka.io achieved up to 3.9 times higher throughput in fault-free scenarios than an implementation of the Raft-based Apache Ratis.

False data injection attacks on networked control systems

The integration of cyber (network) with physical world is a big step in cyber-physical systems. This has revolutionised many industries. But this transformation has made cyber-physical systems vulnerable to attacks. One particular type of attack is the adversarial false data injection which injects false data in either the sensor measurements or the corresponding communication channel. A better understanding of how false data injection (FDI) attacks are constructed is crucial for developing strategies to protect against such attacks. In this paper, we consider two models for networked control systems and present an algorithm for constructing FDI attacks in each case and compare with an existing approach. The conditions for the attack to remain stealthy for systems equipped with a failure detector and the design of attack vectors that satisfy these conditions are discussed in detail. The algorithms are demonstrated by developing FDI attacks for two real-world examples.

Failure Detector Based on Vehicle Movement Prediction in Vehicular Ad-Hoc Networks

Vehicular ad hoc network (VANETs) improves road safety and efficiency by organizing vehicles and infrastructure to provide a platform for application deployment. The availability of vehicles and infrastructure is critical to the operation of applications. Accurate failure detector (FD) has been one of the fundamental components for maintaining high availability in VANETs. However, it is hard to find the vehicle failure accurately and timely due to the dynamic nature of VANETs caused by the high mobility of vehicles and communications link failures. Therefore, it is important to achieve an accurate FD which can cope with the high mobility of VANETs. In this paper, we propose a dead reckoning based FD, called DR-FD. It can predict the mobility of vehicle accurately and avoid the impact of link failures on the detection results by the cooperation between vehicles. Experimental results are provided to confirm that the proposed DR-FD method can achieve at most 20% reduction in detection time, 30% improvement in mistake rate and 20% improvement in overhead.

The missing piece: a distributed system-level diagnosis model for the implementation of unreliable failure detectors

The missing piece: a distributed system-level diagnosis model for the implementation of unreliable failure detectors

Leaderless consensus

Classic synchronous consensus algorithms are leaderless in that processes exchange proposals, choose the maximum value and decide when they see the same choice across a couple of rounds. Indulgent consensus algorithms are typically leader-based. Although they tolerate unexpected delays and find practical applications in blockchains running over an open network like the Internet, their performance is highly dependent on the activity of a single participant. This paper asks whether, under eventual synchrony, it is possible to deterministically solve consensus without a leader. The fact that the weakest failure detector to solve consensus is one that also eventually elects a leader seems to indicate that the answer to the question is negative. We prove in this paper that the answer is actually positive. We first give a precise definition of the very notion of a leaderless algorithm. Then we present three indulgent leaderless consensus algorithms, each we believe interesting in its own right: (i) for shared memory, (ii) for message passing with omission failures and (iii) for message passing with Byzantine failures. Finally, we implement a Byzantine fault tolerant (BFT) state machine replication (SMR), that is leaderless. Our empirical results demonstrate that it is faster and more robust than HotStuff, the recent BFT SMR algorithm used in the Facebook Libra blockchain when deployed in a wide area network.

A case study on parametric verification of failure detectors

Partial synchrony is a model of computation in many distributed algorithms and modern blockchains. These algorithms are typically parameterized in the number of participants, and their correctness requires the existence of bounds on message delays and on the relative speed of processes after reaching Global Stabilization Time. These characteristics make partially synchronous algorithms parameterized in the number of processes, and parametric in time bounds, which render automated verification of partially synchronous algorithms challenging. In this paper, we present a case study on formal verification of both safety and liveness of the Chandra and Toueg failure detector that is based on partial synchrony. To this end, we first introduce and formalize the class of symmetric point-to-point algorithms that contains the failure detector. Second, we show that these symmetric point-to-point algorithms have a cutoff, and the cutoff results hold in three models of computation: synchrony, asynchrony, and partial synchrony. As a result, one can verify them by model checking small instances, but the verification problem stays parametric in time. Next, we specify the failure detector and the partial synchrony assumptions in three frameworks: TLA+, IVy, and counter automata. Importantly, we tune our modeling to use the strength of each method: (1) We are using counters to encode message buffers with counter automata, (2) we are using first-order relations to encode message buffers in IVy, and (3) we are using both approaches in TLA+. By running the tools for TLA+ and counter automata, we demonstrate safety for fixed time bounds. By running IVy, we prove safety for arbitrary time bounds. Moreover, we show how to verify liveness of the failure detector by reducing the verification problem to safety verification. Thus, both properties are verified by developing inductive invariants with IVy.

Secure and Reliable Network Updates

Software-defined wide area networking (SD-WAN) enables dynamic network policy control over a large distributed network via network updates . To be practical, network updates must be consistent (i.e., free of transient errors caused by updates to multiple switches), secure (i.e., only be executed when sent from valid controllers), and reliable (i.e., function despite the presence of faulty or malicious members in the control plane), while imposing only minimal overhead on controllers and switches. We present SERENE: a protocol for se cure and re liable ne twork updates for SD-WAN environments. In short: Consistency is provided through the combination of an update scheduler and a distributed transactional protocol. Security is preserved by authenticating network events and updates, the latter with an adaptive threshold cryptographic scheme. Reliability is provided by replicating the control plane and making it resilient to a dynamic adversary by using a distributed ledger as a controller failure detector. We ensure practicality by providing a mechanism for scalability through the definition of independent network domains and exploiting the parallelism of network updates both within and across domains. We formally define SERENE’s protocol and prove its safety with regards to event-linearizability. Extensive experiments show that SERENE imposes minimal switch burden and scales to large networks running multiple network applications all requiring concurrent network updates, imposing at worst a 16% overhead on short-lived flow completion and negligible overhead on anticipated normal workloads.

On the Properness of Incorporating Binary Classification Machine Learning Algorithms Into Safety-Critical Systems

Manufacturers are willing to incorporate Machine Learning (ML) algorithms into their systems, especially those considered as Safety-Critical Systems (SCS). ML algorithms that perform binary classification (i.e., Binary Classifiers (BCs)) find a wide applicability as error, intrusion or failure detectors, provided that their performance complies with SCS safety requirements. However, the performance analysis of BCs relies on metrics that were not developed with safety in mind and consequently may not provide meaningful evidence to decide whether to incorporate a BC into a SCS. In this paper, we empirically assess the properness of such incorporation by analyzing the distribution of misclassifications of BCs instead of simply counting misclassifications. This allows us to better assess the adequacy of a given BC by identifying areas of the classification space where the BC is likely to misclassify and therefore constitutes actionable information to deal with the SCS. Our assessment takes a deeper view of the classification performance concerning safety by using new metrics that consider the proportions of predictions that are/are not considered sufficiently safe to be used by incorporating SCS. The results of our experiment allow discussing the potential of such distribution analysis for deciding if a BC can be incorporated into a SCS.

Medley: A Membership Service for IoT Networks

Efficient and correct operation of an IoT network requires the presence of a failure detector and membership protocol amongst the IoT nodes. This paper presents a new failure detector for IoT settings wherein nodes are connected via a wireless ad-hoc network. Our failure detector, named Medley, is fully decentralized, allows IoT nodes to maintain a local membership list of other alive nodes, detects failures quickly (and updates the membership list), and incurs low communication overhead. We adapt a failure detector originally proposed for datacenters (SWIM), for the IoT environment. This adaptation is non-trivial. In Medley each node picks a medley of ping targets in a randomized and skewed manner, preferring nearer nodes. We also provide optimizations to achieve time-bounded detection, as well as to reduce tail detection times. Via analysis, simulation, and Raspberry Pi deployments, we show that Medley can simultaneously optimize detection time and communication traffic.

FSNet: A Failure Detection Framework for Semantic Segmentation

Semantic segmentation is an important task that helps autonomous vehicles understand their surroundings and navigate safely. However, during deployment, even the most mature segmentation models are vulnerable to various external factors that can degrade the segmentation performance with potentially catastrophic consequences for the vehicle and its surroundings. To address this issue, we propose a failure detection framework to identify pixel-level misclassification. We do so by exploiting internal features of the segmentation model and training it simultaneously with a failure detection network. During deployment, the failure detector flags areas in the image where the segmentation model has failed to segment correctly. We evaluate the proposed approach against state-of-the-art methods and achieve 12.30%, 9.46%, and 9.65% performance improvement in the AUPR-Error metric for Cityscapes, BDD100k, and Mapillary semantic segmentation datasets.

Data-driven mixed-Integer linear programming-based optimisation for efficient failure detection in large-scale distributed systems

Failure detectors (FDs) are fundamental building blocks for distributed systems. An FD detects whether a process has crashed or not based on the reception of heartbeats’ messages sent by this process over a communication channel. A key challenge of FDs is to tune their parameters to achieve optimal performance which satisfies the desired system requirements. This is challenging due to the complexities of large-scale networks. Existing FDs ignore such optimisation and adopt ad-hoc parameters. In this paper, we propose a new Mixed Integer Linear Programming (MILP) optimisation-based FD algorithm. We obtain the MILP formulation via piecewise linearisation relaxations. The MILP involves obtaining optimal FD parameters that meet the optimal trade-off between its performance metrics requirements, network conditions and system parameters. The MILP maximises our FD’s accuracy under bounded failure detection time while considering network and system conditions as constraints. The MILP’s solution represents optimised FD parameters that maximise FD’s expected performance. To adapt to real-time network changes, our proposed MILP-based FD fits the probability distribution of heartbeats’ inter-arrivals. To address our FD scalability challenge in large-scale systems where the MILP model needs to compute approximate optimal solutions quickly, we also propose a heuristic algorithm. To test our proposed approach, we adopt Amazon Cloud as a realistic testing environment and develop a simulator for robustness tests. Our results show consistent improvement of overall FD performance and scalability. To the best of our knowledge, this is the first attempt to combine the MILP-based optimisation modelling with FD to achieve performance guarantees.

Consensus in anonymous asynchronous systems with crash-recovery and omission failures

In anonymous distributed systems, processes are indistinguishable because they have no identity and execute the same algorithm. Currently, anonymous systems are receiving a lot of attention mainly because they preserve privacy, which is an important property when we want to avoid impersonation attacks. On the other hand, Consensus is a fundamental problem in distributed computing. It is well-known that Consensus cannot be deterministically solved in pure asynchronous anonymous systems if processes can crash (the so-called crash-stop failure model). This impossibility holds even if message losses never occur in transmission. Failure detectors are an elegant and powerful abstraction for achieving deterministic Consensus in asynchronous distributed systems. A failure detector is a distributed object that gives the processes information about crashed processes. Failure detectors have attracted so much attention in the crash-stop failure model because they provide a totally independent abstraction. varOmega is the weakest failure detector to solve Consensus in classic asynchronous systems when a majority of processes never crash, and AvarOmega ' is its implementable version for anonymous systems. As far as we know, there is a lack of works in the literature which tackle Consensus in anonymous asynchronous systems where crashed process can recover (the so-called crash-recovery failure model) and also assuming errors in transmission operations (the so-called omission failure model). Extending failure models in the system allows us to design more realistic systems and solve more practical security problems (i.e., fair exchange and the secure multiparty computation). We present, in this paper, an algorithm to solve Consensus using AvarOmega ' in anonymous asynchronous systems under the crash-recovery and omission failure models. Another important contribution of this paper is a communication-efficient and latency-efficient implementation of AvarOmega ' for these new failure models.

Low-Power Failure Detection for Environmental Monitoring Based on IoT

Many environmental monitoring applications that are based on the Internet of Things (IoT) require robust and available systems. These systems must be able to tolerate the hardware or software failure of nodes and communication failure between nodes. However, node failure is inevitable due to environmental and human factors, and battery depletion in particular is a major contributor to node failure. The existing failure detection algorithms seldom consider the problem of node battery consumption. In order to rectify this, we propose a low-power failure detector (LP-FD) that can provide an acceptable failure detection service and can save on the battery consumption of nodes. From simulation experiments, results show that the LP-FD can provide better detection speed, accuracy, overhead and battery consumption than other failure detection algorithms.