Fault-tolerant Distributed Algorithms Research Articles

Survey on Parameterized Verification with Threshold Automata and the Byzantine Model Checker

Threshold guards are a basic primitive of many fault-tolerant algorithms that solve classical problems in distributed computing, such as reliable broadcast, two-phase commit, and consensus. Moreover, threshold guards can be found in recent blockchain algorithms such as, e.g., Tendermint consensus. In this article, we give an overview of techniques for automated verification of threshold-guarded fault-tolerant distributed algorithms, implemented in the Byzantine Model Checker (ByMC). These threshold-guarded algorithms have the following features: (1) up to $t$ of processes may crash or behave Byzantine; (2) the correct processes count messages and make progress when they receive sufficiently many messages, e.g., at least $t+1$; (3) the number $n$ of processes in the system is a parameter, as well as the number $t$ of faults; and (4) the parameters are restricted by a resilience condition, e.g., $n > 3t$. Traditionally, these algorithms were implemented in distributed systems with up to ten participating processes. Nowadays, they are implemented in distributed systems that involve hundreds or thousands of processes. To make sure that these algorithms are still correct for that scale, it is imperative to verify them for all possible values of the parameters.

Adaptive Fault-Tolerant Distributed Formation Control of Clustered Vertical Takeoff and Landing UAVs

This article investigates the formation control issue of clustered vertical takeoff and landing (VTOL) unmanned airborne vehicles (UAVs) subject to undesired perturbations composed by actuator faults, parametric uncertainties, and external disturbances. In particular, a cluster of VTOL UAVs fly together in a given velocity with a well-specified formation configuration. For this purpose, an adaptive fault-tolerant distributed control algorithm via local information interaction is proposed under a hierarchical framework. Specifically, a distributed command force resorting to projection-based adaptation is first synthesized in the outer position loop. Such a command force is of a saturation attribute in order that a boundeddesired thrust and a nonsingular command attitude can be extracted in terms of appropriate control parameters. Next, in the inner attitude loop, a desired torque also with the help of projection-based adaptation is synthesized. Given that the underlying undirected topology is connected, it is shown based on the Lyapunov theory that the proposed adaptive fault-tolerant distributed control algorithm ensures the accomplishment of the concerned formation control objective of the clustered VTOL UAVs in spite of the undesired perturbations. Simulations validate and further highlight the effectiveness of the proposed distributed control algorithm.

Verifying safety of synchronous fault-tolerant algorithms by bounded model checking

Threshold automata are a formalism introduced for modeling, verification, and synthesis of fault-tolerant distributed algorithms for asynchronous systems, that is, in interleaving semantics. Owing to well-known limitations of what can be achieved in purely asynchronous systems, many fault-tolerant distributed algorithms are designed for synchronous or round-based semantics. In this paper, we introduce the synchronous variant of threshold automata and study their applicability and limitations for the verification of synchronous fault-tolerant distributed algorithms. We show that the parameterized reachability problem for synchronous threshold automata is undecidable. Still, we show that many synchronous fault-tolerant distributed algorithms have a bounded diameter, even though the algorithms are parameterized by the number of processes. Hence, bounded model checking can be used for verifying these algorithms. The existence of bounded diameters is the main conceptual insight in this paper. We compute the diameter of several algorithms and check their safety properties, using SMT queries that contain quantifiers for dealing with the parameters symbolically. Surprisingly, performance of the SMT solvers on these queries is very good, reflecting the recent progress in dealing with quantified queries. We found that the diameter bounds of synchronous algorithms in the literature are tiny (from 1 to 8), which makes our approach applicable in practice. For a specific class of algorithms, we also establish a theoretical result on the existence of a diameter, providing a first explanation for our experimental results.

Verification of randomized consensus algorithms under round-rigid adversaries

Randomized fault-tolerant distributed algorithms pose a number of challenges for automated verification: (i) parameterization in the number of processes and faults, (ii) randomized choices and probabilistic properties, and (iii) an unbounded number of asynchronous rounds. This combination makes verification hard. Challenge (i) was recently addressed in the framework of threshold automata. We extend threshold automata to model randomized consensus algorithms that perform an unbounded number of asynchronous rounds. For non-probabilistic properties, we show that it is necessary and sufficient to verify these properties under round-rigid schedules, that is, schedules where processes enter round r only after all processes finished round r-1. For almost-sure termination, we analyze these algorithms under round-rigid adversaries, that is, fair adversaries that only generate round-rigid schedules. This allows us to do compositional and inductive reasoning that reduces verification of the asynchronous multi-round algorithms to model checking of a one-round threshold automaton. We apply this framework and automatically verify the following classic algorithms: Ben-Or’s and Bracha’s seminal consensus algorithms for crashes and Byzantine faults, 2-set agreement for crash faults, and RS-Bosco for the Byzantine case.

Randomized proof-labeling schemes

Proof-labeling schemes, introduced by Korman et al. (Distrib Comput 22(4):215–233, 2010. https://doi.org/10.1007/s00446-010-0095-3 ), are a mechanism to certify that a network configuration satisfies a given boolean predicate. Such mechanisms find applications in many contexts, e.g., the design of fault-tolerant distributed algorithms. In a proof-labeling scheme, predicate verification consists of neighbors exchanging labels, whose contents depends on the predicate. In this paper, we introduce the notion of randomized proof-labeling schemes where messages are randomized and correctness is probabilistic. We show that randomization reduces verification complexity exponentially while guaranteeing probability of correctness arbitrarily close to one. We also present a novel message-size lower bound technique that applies to deterministic as well as randomized proof-labeling schemes. Using this technique, we establish several tight bounds on the verification complexity of MST, acyclicity, connectivity, and longest cycle size.

Metastability-Containing Circuits

In digital circuits, metastability can cause deteriorated signals that neither are logical 0 nor logical 1, breaking the abstraction of Boolean logic. Synchronizers, the only traditional countermeasure, exponentially decrease the odds of maintained metastability over time. We propose a fundamentally different approach: It is possible to deterministically contain metastability by fine-grained logical masking so that it cannot infect the entire circuit. At the heart of our approach lies a time- and value-discrete model for metastability in synchronous clocked digital circuits, in which metastability is propagated in a worst-case fashion. The proposed model permits positive results and passes the test of reproducing Marino's impossibility results. We fully classify which functions can be computed by circuits with standard registers. Regarding masking registers, we show that more functions become computable with each clock cycle, and that masking registers permit exponentially smaller circuits for some tasks. Demonstrating the applicability of our approach, we present the first fault-tolerant distributed clock synchronization algorithm that deterministically guarantees correct behavior in the presence of metastability. As a consequence, clock domains can be synchronized without using synchronizers, enabling metastability-free communication between them.

Energy efficient and fault tolerant distributed algorithm for data aggregation in wireless sensor networks

Mobile Agent (MA) paradigm for data aggregation in wireless sensor networks (WSNs) presents a distributed computing paradigm which has proved its efficiency in comparison to the traditional client–server computing paradigm. In terms of energy consumption and overall time response, MA computing paradigm presents a better alternative. Instead of sending the collected data to the sink as in client/server, MA moves to sensor nodes (SNs) for data collection. For MA, to move among SNs, an itinerary should be planned before the migration. Many approaches have been proposed to solve the problem, but all approaches did not take into consideration the fault tolerance problem, even though WSNs are prone to failure. In this respect, we propose Energy efficient and fault tolerant distributed algorithm for data aggregation in wireless sensor networks (EFTA) to plan the itinerary for MA and another alternative itinerary in case of node(s) failure. Our algorithm adopts a clustering method to group SNs in clusters then plans the itinerary among those clusters efficiently. Also an alternative itinerary based fault tolerance is proposed. Simulation results show that our algorithm performs better than other existing ones.

Para^2: parameterized path reduction, acceleration, and SMT for reachability in threshold-guarded distributed algorithms

Automatic verification of threshold-based fault-tolerant distributed algorithms (FTDA) is challenging: FTDAs have multiple parameters that are restricted by arithmetic conditions, the number of processes and faults is parameterized, and the algorithm code is parameterized due to conditions counting the number of received messages. Recently, we introduced a technique that first applies data and counter abstraction and then runs bounded model checking (BMC). Given an FTDA, our technique computes an upper bound on the diameter of the system. This makes BMC complete for reachability properties: it always finds a counterexample, if there is an actual error. To verify state-of-the-art FTDAs, further improvement is needed. In contrast to encoding bounded executions of a counter system over an abstract finite domain in SAT, in this paper, we encode bounded executions over integer counters in SMT. In addition, we introduce a new form of reduction that exploits acceleration and the structure of the FTDAs. This aggressively prunes the execution space to be explored by the solver. In this way, we verified safety of seven FTDAs that were out of reach before.

Fault-Tolerant Rendezvous of Multirobot Systems

In this paper, we propose a distributed control policy to achieve rendezvous by a set of robots even when some robots in the system do not follow the prescribed policy. These nonconforming robots correspond to faults in the multirobot system, and our control policy is thus a fault-tolerant policy. Each robot has a limited sensing range and is able to directly estimate the state of only those robots within that sensing range, which induces a network topology for the multirobot system. We assume that it is not possible for the fault-free robots to identify the faulty robots, and thus our approach is robust even to undetected faults in the system. The main contribution of this paper is a fault-tolerant distributed control algorithm that is guaranteed to converge to consensus under certain reasonable connectivity conditions. We first present a general algorithm that exploits the notion of a Tverberg partition of a point set in $\mathbb {R}^d$ , and give a proof of convergence. We then provide three instantiations of this algorithm, based on three different sensing models. For each case, we analyze performance via extensive simulations. The effectiveness and performance of our algorithms on real platforms are demonstrated through experiments on a multirobot testbed.

Cardinality constraints for arrays (decidability results and applications)

Enriching logic formalisms with counting capabilities is an important task in view of the needs of many application areas, ranging from database theory to formal verification. In this paper, we consider a very expressive language obtained by enriching linear integer arithmetic with free function symbols and cardinality constraints for interpreted sets. We obtain positive results for a flat fragment via a reduction to decidability of Presburger arithmetic with unary counting quantifiers (Schweikhart in Arithmetic, first-order logic, and counting quantifiers, ACM TOCL, New York, 2004). We isolate also an easier simple flat subfragment, whose satisfiability is in NP, and we show that this subfragment is adequate to formalize problems arising in the area of the verification of fault-tolerant distributed algorithms. We finally discuss our first implementation, the related experimental results, as well as further algorithmic problems suggested by model-checking applications.

A short counterexample property for safety and liveness verification of fault-tolerant distributed algorithms

Distributed algorithms have many mission-critical applications ranging from embedded systems and replicated databases to cloud computing. Due to asynchronous communication, process faults, or network failures, these algorithms are difficult to design and verify. Many algorithms achieve fault tolerance by using threshold guards that, for instance, ensure that a process waits until it has received an acknowledgment from a majority of its peers. Consequently, domain-specific languages for fault-tolerant distributed systems offer language support for threshold guards. We introduce an automated method for model checking of safety and liveness of threshold-guarded distributed algorithms in systems where the number of processes and the fraction of faulty processes are parameters. Our method is based on a short counterexample property: if a distributed algorithm violates a temporal specification (in a fragment of LTL), then there is a counterexample whose length is bounded and independent of the parameters. We prove this property by (i) characterizing executions depending on the structure of the temporal formula, and (ii) using commutativity of transitions to accelerate and shorten executions. We extended the ByMC toolset (Byzantine Model Checker) with our technique, and verified liveness and safety of 10 prominent fault-tolerant distributed algorithms, most of which were out of reach for existing techniques.

On the completeness of bounded model checking for threshold-based distributed algorithms: Reachability

Counter abstraction is a powerful tool for parameterized model checking, if the number of local states of the concurrent processes is relatively small. In recent work, we introduced parametric interval counter abstraction that allowed us to verify the safety and liveness of threshold-based fault-tolerant distributed algorithms (FTDA). Due to state space explosion, applying this technique to distributed algorithms with hundreds of local states is challenging for state-of-the-art model checkers. In this paper, we demonstrate that reachability properties of FTDAs can be verified by bounded model checking. To ensure completeness, we need an upper bound on the distance between states. We show that the diameters of accelerated counter systems of FTDAs, and of their counter abstractions, have a quadratic upper bound in the number of local transitions. Our experiments show that the resulting bounds are sufficiently small to use bounded model checking for parameterized verification of reachability properties of several FTDAs, some of which have not been automatically verified before.

PSync: a partially synchronous language for fault-tolerant distributed algorithms

Fault-tolerant distributed algorithms play an important role in many critical/high-availability applications. These algorithms are notoriously difficult to implement correctly, due to asynchronous communication and the occurrence of faults, such as the network dropping messages or computers crashing. We introduce PSync, a domain specific language based on the Heard-Of model, which views asynchronous faulty systems as synchronous ones with an adversarial environment that simulates asynchrony and faults by dropping messages. We define a runtime system for PSync that efficiently executes on asynchronous networks. We formalise the relation between the runtime system and PSync in terms of observational refinement. The high-level lockstep abstraction introduced by PSync simplifies the design and implementation of fault-tolerant distributed algorithms and enables automated formal verification. We have implemented an embedding of PSync in the Scala programming language with a runtime system for partially synchronous networks. We show the applicability of PSync by implementing several important fault-tolerant distributed algorithms and we compare the implementation of consensus algorithms in PSync against implementations in other languages in terms of code size, runtime efficiency, and verification.

Fault-tolerant algorithms for tick-generation in asynchronous logic

Today’s hardware technology presents a new challenge in designing robust systems. Deep submicron VLSI technology introduces transient and permanent faults that were never considered in low-level system designs in the past. Still, robustness of that part of the system is crucial and needs to be guaranteed for any successful product. Distributed systems, on the other hand, have been dealing with similar issues for decades. However, neither the basic abstractions nor the complexity of contemporary fault-tolerant distributed algorithms match the peculiarities of hardware implementations. This article is intended to be part of an attempt striving to bridge over this gap between theory and practice for the clock synchronization problem. Solving this task sufficiently well will allow to build an ultra-robust high-precision clocking system for hardware designs like systems-on-chips in critical applications. As our first building block, we describe and prove correct a novel distributed, Byzantine fault-tolerant, probabilistically self-stabilizing pulse synchronization protocol, called FATAL, that can be implemented using standard asynchronous digital logic: Correct FATAL nodes are guaranteed to generate pulses (i.e., unnumbered clock ticks) in a synchronized way, despite a certain fraction of nodes being faulty. FATAL uses randomization only during stabilization and, despite the strict limitations introduced by hardware designs, offers optimal resilience and smaller complexity than all existing protocols. Finally, we show how to leverage FATAL to efficiently generate synchronized, self-stabilizing, high-frequency clocks.

Reconciling fault-tolerant distributed algorithms and real-time computing.

We present generic transformations, which allow to translate classic fault-tolerant distributed algorithms and their correctness proofs into a real-time distributed computing model (and vice versa). Owing to the non-zero-time, non-preemptible state transitions employed in our real-time model, scheduling and queuing effects (which are inherently abstracted away in classic zero step-time models, sometimes leading to overly optimistic time complexity results) can be accurately modeled. Our results thus make fault-tolerant distributed algorithms amenable to a sound real-time analysis, without sacrificing the wealth of algorithms and correctness proofs established in classic distributed computing research. By means of an example, we demonstrate that real-time algorithms generated by transforming classic algorithms can be competitive even w.r.t. optimal real-time algorithms, despite their comparatively simple real-time analysis.

Delay Sensitive, Energy Efficient and Fault Tolerant Distributed Slot Assignment Algorithm for Wireless Sensor Networks under Convergecast Data Traffic

The scarcest resource for most of the wireless sensor networks (WSNs) is energy and the major factor in energy consumption for WSNs is communication. Not only transmission, but also reception is the source of energy consumption. The lore to decrease energy consumption is to turn off the radio circuit when it is not needed. This is why TDMA is advantageous over contention based methods. A time slot assignment algorithm is an essential part of TDMA based systems. Although centralized time slot assignment protocols are preferred in many WSNs, a centralized approach is not scalable. In this article, a new energy efficient, delay sensitive, and fault tolerant distributed time slot assignment algorithm, referred to as ft_DTSM, is proposed for sensor networks under convergecast traffic pattern. ft_DTSM aims at operating with low delay and low energy under faulty nodes assumption. Instead of random access based periods, it assigns slots with the help of tiny request slots. While traditional slot assignment algorithms do not allow assigning the same slot within two hop neighbors, because of the hidden node problem, ft_DTSM can assign, if the assignment is suitable for convergecast traffic. Simulation results have shown that both delay and energy consumption performances of ft_DTSM is superior to existing distributed slot assignment protocols for WSNs. ft_DTSM can also distribute the slots so that the network can continue its operation against a single point of failure. Although ft_DTSM has a somewhat longer execution time, its scalability characteristic may provide application specific time durations.

Automatic Verification of Fault-Tolerant Register Emulations

The design and verification of fault-tolerant distributed algorithms is a complicated task. Usually, the proof of correctness is done manually, and thus depends on the skill of the prover. Using automated verification methods, such as model checking, can greatly simplify the verification. However, model checking of distributed algorithms is often intractable because of the state-explosion problem. In this paper we present a novel approach to verification of quorum-based distributed register emulation algorithms with undetectable crash failures of processes. Our approach is based on projection and abstraction and allows us to reduce the task of model-checking the whole system to fair model-checking of subsystems consisting of a constant number of processes. Our method is highly scalable and can be applied to a large class of algorithms. Aside from efficient verification, it can also be used for finding redundancies in existing algorithms.

A Self-Stabilizing Approximation Algorithm for the Distributed Minimum k-Domination

Self-stabilization is a theoretical framework of nonmasking fault-tolerant distributed algorithms. In this paper, we investigate a self-stabilizing distributed approximation for the minimum k-dominating set (KDS) problem in general networks. The minimum KDS problem is a generalization of the well-known dominating set problem in graph theory. For a graph G = (V, E), a set Dk ⊆ V is a KDS of G if and only if each vertex not in Dk is adjacent to at least k vertices in Dk. The approximation ratio of our algorithm is Δ/k(1 + (k-1/Δ+1)), where Δ is the maximum degree of G, in the networks of which the minimum degree is more than or equal to k.

A short introduction to failure detectors for asynchronous distributed systems

Since the first version of Chandra and Toueg's seminal paper titled "Unreliable failure detectors for reliable distributed systems" in 1991, the failure detector concept has been extensively studied and investigated. This is not at all surprising as failure detection is pervasive in the design, the analysis and the implementation of a lot of fault-tolerant distributed algorithms that constitute the core of distributed system middleware.The literature on this topic is mostly technical and appears mainly in theoretically inclined journals and conferences. The aim of this paper is to offer an introductory survey to the failure detector concept for readers who are not familiar with it and want to quickly understand its aim, its basic principles, its power and limitations. To attain this goal, the paper first describes the motivations that underlie the concept, and then surveys several distributed computing problems showing how they can be solved with the help of an appropriate failure detector. So, this short paper presents motivations, concepts, problems, definitions, and algorithms. It does not contain proofs. It is aimed at people who want to understand basics of failure detectors.

A mutual exclusion algorithm for ad hoc mobile networks

A fault-tolerant distributed mutual exclusion algorithm that adjusts to node mobility is presented, along with proof of correctness and simulation results. The algorithm requires nodes to communica...