Crash Faults Research Articles

Achieving Crash Fault Tolerance In Autonomous Vehicle Autopilot Software Stacks Through Safety-Critical Module Rejuvenation

Autonomous driving vehicles (ADV), have been in recent years, victims of their own success. Through their use of increasingly sophisticated sensor modalities and deep learning capabilities, ADVs have not only learned how to probe their chaotic environment with higher granularity coupled with smooth trajectory execution but also inherited all the vulnerabilities that were hiding behind these new features. Ensuring the safety of ADVs is crucial since a simple fault along their underlying autopilot software stack can lead to catastrophic accidents with the loss of human lives. Therefore we propose a crash-fault tolerant scheme that can be triggered whenever a crash fault of the safety critical submodules of the autopilot software stack is detected, which executes an emergency trajectory and effectively steers the car into a safe spot where the autopilot can be rejuvenated. We implement and evaluate the efficacy of this recovery scheme in the Apollo ADV software stack in conjunction with the SVL simulator. Keywords: autonomous driving vehicles, rejuvenation, crash fault tolerance, simplex architecture, apollo software stack.

Scalable atomic broadcast: A leaderless hierarchical algorithm

Atomic Broadcast is an essential broadcast primitive as it ensures the consistency of distributed replicas. However, it is notoriously non-scalable. In this work, we introduce the Leaderless Hierarchical Atomic Broadcast (LHABcast) algorithm, which has two properties to improve scalability. First, it is a fully decentralized algorithm that does not rely on a sequencer/leader, which is often a significant bottleneck. Processes running LHABcast send messages with local sequence numbers and order messages received from other processes using timestamps inspired on Lamport's logical clocks. A process that receives the required set of timestamps can make a decision about the overall sequence of message delivery. Second, the algorithm is hierarchical: processes are organized on a vCube logical overlay network, which has several logarithmic properties and allows the construction of autonomous spanning trees. vCube also works as a failure detector, assuming crash faults and an asynchronous system model. In this paper, LHABcast is described, specified, and proven to be correct. Both simulation and experimental results are presented. A comparison with an all-to-all strategy shows that the number of messages sent by LHABcast is significantly lower in both fault-free and faulty scenarios. An implementation of LHABcast in Akka.io achieved up to 3.9 times higher throughput in fault-free scenarios than an implementation of the Raft-based Apache Ratis.

On the performance and scalability of consensus mechanisms in privacy-enabled decentralized renewable energy marketplace

Renewable energy sources were introduced as an alternative to fossil fuel sources to make electricity generation cleaner. However, today’s renewable energy markets face a number of limitations, such as inflexible pricing models and inaccurate consumption information. These limitations can be addressed with a decentralized marketplace architecture. Such architecture requires a mechanism to guarantee that all marketplace operations are executed according to predefined rules and regulations. One of the ways to establish such a mechanism is blockchain technology. This work defines a decentralized blockchain-based peer-to-peer (P2P) energy marketplace which addresses actors’ privacy and the performance of consensus mechanisms. The defined marketplace utilizes private permissioned Ethereum-based blockchain client Hyperledger Besu (HB) and its smart contracts to automate the P2P trade settlement process. Also, to make the marketplace compliant with energy trade regulations, it includes the regulator actor, which manages the issue and consumption of guarantees of origin and certifies the renewable energy sources used to generate traded electricity. Finally, the proposed marketplace incorporates privacy-preserving features, allowing it to generate private transactions and store them within a designated group of actors. Performance evaluation results of HB-based marketplace with three main consensus mechanisms for private networks, i.e., Clique, IBFT 2.0, and QBFT, demonstrate a lower throughput than another popular private permissioned blockchain platform Hyperledger Fabric (HF). However, the lower throughput is a side effect of the Byzantine Fault Tolerant characteristics of HB’s consensus mechanisms, i.e., IBFT 2.0 and QBFT, which provide increased security compared to HF’s Crash Fault Tolerant consensus RAFT.

RAFT Consensus Reliability in Wireless Networks: Probabilistic Analysis

The centralized system becomes less efficient, secure, and resilient as the network size and heterogeneity increase due to its inherent single point of failure issues. Distributed consensus mechanisms characterized by decentralization, autonomy, parallelism and fault-tolerance can meet the increasing demands of safety and security in critical interconnected systems. This paper establishes a Node and Link probabilistic failure model in the presence of node and communication link failures for a representative crash fault tolerant distributed consensus protocol: RAFT. The analytical results in terms of the probability density function and the mean value of consensus reliability are derived. Two important reliability performance indicators, Reliability Gain and Tolerance Gain are proposed to indicate the linear relationship between the consensus reliability and two basic parameters, i.e. the joint failure rate and the maximum number of tolerant faulty nodes, which provide the theoretical guidance for quickly deploying a RAFT system. The special case of a distributed consensus network with already a certain number of failures and its adverse impact are evaluated. The Markov probabilistic models, definitions of Reliability Gain and Tolerance Gain, and the analysis methods proposed in this paper can be extended to other consensus mechanisms.

Improving Trust in a (Trans)National Invoicing System: The Performance of Crash vs. Byzantine Fault Tolerance at Scale

Crash fault tolerance describes the capability of a distributed system to maintain its proper function despite the occurrence of crashes or failures in one or more of its components. When a distributed system possesses crash fault tolerance, it can be further fortified to achieve Byzantine fault tolerance. Byzantine fault tolerance empowers a distributed system to establish consensus among participants, even when faced with faulty or malicious behavior. Consensus plays a critical role in various tasks, including determining the accurate value of a shared variable, electing a leader, or validating the integrity of a business transaction. Compared to crash fault tolerance, Byzantine fault tolerance instills greater trust because it enables consensus even in the presence of malicious entities. This paper focuses on the performance evaluation of two blockchain solutions that exhibit Byzantine fault tolerance, in contrast to a blockchain solution that demonstrates crash fault tolerance. Specifically, the paper investigates the additional performance requirements associated with the enhanced trust resulting from Byzantine fault tolerance in e-business trading on both national and transnational scales. We analyze the resources needed to operate a business-to-business/business-to-government (B2B/B2G) compliance framework in two distinct geographic scenarios. The first examines the national scale, using Denmark as an example, which is the eleventh largest European country by GDP. The second scenario considers the scale of the European Union (EU) with its 27 member states (plus the United Kingdom).

Fault tolerant network constructors

We consider adversarial crash faults of nodes in the network constructors model [Michail and Spirakis, 2016]. We first show that, without further assumptions, the class of graph languages that can be (stably) constructed under crash faults is non-empty but small. On the positive side, linear waste enables the construction, on a fraction of the nodes, of any graph language that is constructible in the fault-free case and partial constructibility allows us to construct a large class of graph languages. We then extend the original model with a minimal form of fault notifications. Our main result under that model is a fault-tolerant universal constructor. Finally, we show that logarithmic local memories can be exploited for a no-waste fault-tolerant simulation of any network constructor.

Byzantine fault tolerant symmetric-persistent circle evacuation

We consider (n,f)-evacuation from a disk, a problem in which n>1 robots, f of which are faulty, seek to evacuate through a hidden exit which is located on the perimeter of a unit disk.We focus on symmetric-persistent algorithms, a common natural approach to search and evacuation problems. We consider two communication models: wireless and face-to-face. For the wireless model we first prove a lower bound of 1+4πn+2sin⁡(π2−πn) for the case of one faulty robot. We also observe an almost matching upper bound obtained by utilizing an earlier search algorithm. We then study the case with two Byzantine robots and we provide an algorithm that achieves evacuation in time at most 3+6πn+δ(n), where δ(n) is a decreasing function with maximum value δ(4)=0.5687, vanishing for n≥9. For the face-to-face model we provide an upper bound of 3+(f+1)2πn for evacuation of n robots under crash faults, an upper bound of 3+4πn+2sin⁡πn for evacuation in the case of one Byzantine robot and an upper bound of 3+6πn+2sin⁡2πn in the case of two Byzantine robots.

Optimal circle search despite the presence of faulty robots

We consider (n,f)-search on a circle, a search problem of a hidden exit on a circle of unit radius for n>1 robots, f of which are faulty. All the robots start at the center of the circle and can move anywhere with maximum speed 1. During the search, robots may communicate wirelessly. All messages transmitted by all robots are tagged with the robots' unique identifiers which cannot be corrupted. The search is considered complete when the exit is found by a non-faulty robot (which must visit its location) and the remaining non-faulty robots know the correct location of the exit.We study two models of faulty robots. First, crash-faulty robots may stop operating as instructed, and thereafter they remain nonfunctional. Second, Byzantine-faulty robots may transmit untrue messages at any time during the search so as to mislead the non-faulty robots, e.g., lie about the location of the exit.When there are only crash fault robots, we provide optimal algorithms for the (n,f)-search problem, with optimal worst-case search completion time 1+(f+1)2πn. Our main technical contribution pertains to optimal algorithms for (n,1)-search with a Byzantine-faulty robot, minimizing the worst-case search completion time, which equals 1+4πn.We also present an algorithm for the mixed case, with one Byzantine and f−1 crash faulty robots with worst-case search completion time 1+2πnf+2sin⁡2πn.

A comparative study on consensus mechanism with security threats and future scopes: Blockchain

Recently, blockchain has emerged as a peer-to-peer (P2P) immutable distributed ledger technology-based network, and its consensus mechanism is playing an important role in managing decentralized data. The use of the consensus process for open blockchains, on the other hand, has revitalized the industry and spurred new architectures. As part of this research, we have analyzed and categorized a wide range of consensus mechanisms in order to contribute to the process of developing blockchain applications and determining the extent of their influence. Critical technical considerations regarding performance, scalability, and other quality aspects that a blockchain-based system must possess may be assisted by this scientific categorization and comprehensive comparison of consensus algorithms. In this paper, we have mainly depicted: (i) proof-of-work (PoW) along with its enhancement rules like greedy heaviest-observed sub-tree (GHOST) and Bitcoin NG (next generation); (ii) proof-of-Stake (PoS) along with chain-based PoS, committee-based PoS, byzantine fault tolerance (BFT) based extended proof-of-stake (EPoS), and delegated proof of stake (DPoS), i.e. energy-efficient alternative to PoW; (iii) BFT consensus to handle crash fault and Byzantine fault; and (iv) hybrid protocols, which are a combination of the best features of PoW, PoS, and BFT consensus mechanism. To evaluate the performance, scalability, security, and design properties of the system, we have considered five components of consensus algorithms.

Failure analysis of unmanned autonomous swarm considering cascading effects

In this paper, we focus on the failure analysis of unmanned autonomous swarm (UAS) considering cascading effects. A framework of failure analysis for UAS is proposed. Guided by the framework, the failure analysis of UAS with crash fault agents is performed. Resilience is used to analyze the processes of cascading failure and self-repair of UAS. Through simulation studies, we reveal the pivotal relationship between resilience, the swarm size, and the percentage of failed agents. The simulation results show that the swarm size does not affect the cascading failure process but has much influence on the process of self-repair and the final performance of the swarm. The results also reveal a tipping point exists in the swarm. Mean-while, we get a counter-intuitive result that larger-scale UAS loses more resilience in the case of a small percentage of failed individuals, suggesting that the increasing swarm size does not necessarily lead to high resilience. It is also found that the temporal degree failure strategy performs much more harmfully to the resilience of swarm systems than the random failure. Our work can provide new insights into the mechanisms of swarm collapse, help build more robust UAS, and develop more efficient failure or protection strategies.

A comprehensive investigation of the impact of feature selection techniques on crashing fault residence prediction models

Software crash is a serious form of the software failure, which often occurs during the software development and maintenance process. As the stack trace reported when the software crashes contains a wealth of information about crashes, recent work utilized classification models with the collected features from stack traces and source code to predict whether the fault causing the crash resides in the stack trace. This could speed-up the crash localization task. As the quality of features can affect the performance of the constructed classification models, researchers proposed to use feature selection methods to select a representative feature subset to build models by replacing the original features. However, only limited feature selection methods and classification models were taken into consideration for this issue in previous work. In this work, we look into this topic deeply and find out the best feature selection method for crash fault residence prediction task. We study the performance of 24 feature selection techniques with 21 classification models on a benchmark dataset containing crash instances from 7 real-world software projects. We use 4 indicators to evaluate the performance of these feature selection methods which are applied to the classification models. The experimental results show that, overall, a probability-based feature selection, called Symmetrical Uncertainty, performs well across the studied classification models and projects. Thus, we recommend such a feature selection method to preprocess the crash instances before constructing classification models to predict the crash fault residence. This work conducts a large-scale empirical study to investigate the impact of feature selection methods on the performance of classification models for the crashing fault residence prediction task. The results clearly demonstrate that there exist significant performance differences among these feature selection techniques across different classification models and projects.

SmartStream

Data streaming platforms connect heterogeneous services through the publish-subscribe paradigm. Currently available platforms provide protection against crash faults, but are not resistant against Byzantine faults like arbitrary hardware faults and intrusions. State machine replication can provide this protection, but the higher resource requirements and the more elaborate communication primitives usually result in a higher overall complexity and a non-negligible performance degradation. As data streaming operates on highly-partitionable append-only state, some of these performance losses can be counteracted by applying speculative execution and sharding. We show the effectiveness of these concepts in a prototype implementation, which only results in a reasonable drop in system throughput and latency during average system utilization, when compared to state-of-the-art data streaming platforms like Apache Kafka, while providing stronger resilience guarantees.

Verification of randomized consensus algorithms under round-rigid adversaries

Randomized fault-tolerant distributed algorithms pose a number of challenges for automated verification: (i) parameterization in the number of processes and faults, (ii) randomized choices and probabilistic properties, and (iii) an unbounded number of asynchronous rounds. This combination makes verification hard. Challenge (i) was recently addressed in the framework of threshold automata. We extend threshold automata to model randomized consensus algorithms that perform an unbounded number of asynchronous rounds. For non-probabilistic properties, we show that it is necessary and sufficient to verify these properties under round-rigid schedules, that is, schedules where processes enter round r only after all processes finished round r-1. For almost-sure termination, we analyze these algorithms under round-rigid adversaries, that is, fair adversaries that only generate round-rigid schedules. This allows us to do compositional and inductive reasoning that reduces verification of the asynchronous multi-round algorithms to model checking of a one-round threshold automaton. We apply this framework and automatically verify the following classic algorithms: Ben-Or’s and Bracha’s seminal consensus algorithms for crashes and Byzantine faults, 2-set agreement for crash faults, and RS-Bosco for the Byzantine case.

Modeling of Two-Level Checkpointing With Silent and Fail-Stop Errors in Grid Computing Systems

With the increase in high-performance computing platform size, it makes the system reliability more challenging, and system mean time between failures (MTBF) may be too short to supply a total fault-free run. Thereby, to achieve greater benefit from these systems, the applications must include fault tolerance mechanisms to satisfy the required reliability. This manuscript focuses on grid computing platform that exposes to two types of threats: crash and silent data corruption faults, which cause the application's failure. This manuscript also addresses the problem of modeling resource availability and aims to minimize the overhead of checkpoint/recovery-fault tolerance techniques. Modeling resources faults has commonly been addressed with exponential distribution, but that isn't fully realistic for the transient errors, which appear randomly. In the manuscript, the authors use Weibull distribution to express these random faults to create the optimal time to save checkpoints.

Fault-tolerant complete visibility for asynchronous robots with lights under one-axis agreement

We consider the distributed setting of N autonomous mobile robots that operate in Look-Compute-Move (LCM) cycles and communicate with other robots using colored lights following the robots with lights model. We assume obstructed visibility under which a robot cannot see another robot if a third robot is positioned between them on the straight line connecting them. We study the fundamental Complete Visibility problem of repositioning N robots, starting from N distinct points, on a plane so that each robot is visible to all others. We are interested in fault-tolerant algorithms. We study fault-tolerance with respect to failures on the mobility of robots. Therefore, any algorithm for Complete Visibility is required to provide visibility between all non-faulty robots (i.e., no three non-faulty robots are collinear and no faulty robot is between two non-faulty robots on the straight line connecting them), independently of the behavior of the faulty ones. We model mobility failures as crash faults in which each faulty robot may stop its movement at any time and, once it stopped moving, it will remain stationary indefinitely thereafter. There exists an algorithm for this problem that tolerates a single faulty robot in the semi-synchronous setting under both-axis agreement. In that algorithm, the light of the faulty robot does not need to work correctly after the robot experiences fault. In this article, we assume the model in which, even after a robot experiences fault, its light still operates correctly and provide the first algorithm for Complete Visibility that tolerates f≤N faulty robots in the asynchronous setting under one-axis agreement. The proposed algorithm has many interesting properties.

Fireplug: Efficient and Robust Geo-Replication of Graph Databases

Although graph-databases have been assuming an increasing relevance in applications that exhibit strong dependability requirements, including tolerance to malicious faults, few works have addressed Byzantine fault tolerance in this particular context, and previous attempts suffer from lack of flexibility and poor performance. This article describes and evaluates Fireplug, a flexible architecture to build robust geo-replicated graph databases. Fireplug can be configured to tolerate from crash to Byzantine faults, both within and across different datacenters. Furthermore, Fireplug is robust to bugs in existing graph database implementations, as it allows to combine multiple graph database instances in a cohesive manner. Thus, Fireplug can support many different deployments, according to the performance/robustness trade-offs imposed by the target application. Our evaluation shows that Fireplug is able implement Byzantine fault tolerance without penalty when compared to the built-in replication mechanism of Neo4j, which only supports crash faults. Additionally, performance optimizations introduced by Fireplug improve the overall performance by up to 900 percent in geo-replicated scenarios.

Lightweight approach to automated fault diagnosis in WSNs

Automated fault diagnosis in wireless sensor networks (WSNs) has always been an essential field for the networks research community. In resource-related constraints, the design of a lightweight method to automatically diagnose the fault in WSN is highly beneficial. In this study, the authors propose lightweight and less-overhead approaches to diagnose hard and soft faults in WSNs automatically. Correctly, a lightweight checksum method is implemented for hard or crash fault detection. This method is capable of detecting multiple hard faults within a single path with the help of a timeout mechanism. For diagnosis of soft faults such as permanent, intermittent, and transient faults, they implement the Anderson–Darling (AD) statistical method. The AD test analyses how the sensor readings are fitted in a specific distribution for a tested significance level. Several testbed experiments are conducted to validate their hypotheses and implementations. The performance parameters that are evaluated in this work include fault detection accuracy, false alarm rate, and false-positive rate. These parameters are evaluated about various fault probabilities in the network. They observe that the proposed lightweight schemes can diagnose both hard and soft faults in message complexity over the network, which makes it adaptable in practise.

Proofs of Writing for Robust Storage

Existing Byzantine fault tolerant (BFT) storage solutions that achieve strong consistency and high availability, are costly compared to solutions that tolerate simple crashes. This cost is one of the main obstacles in deploying BFT storage in practice. In this paper, we present PoWerStore, a robust and efficient data storage protocol. PoWerStore's robustness comprises tolerating network outages, maximum number of Byzantine storage servers, any number of Byzantine readers and crash-faulty writers, and guaranteeing high availability (wait-freedom) and strong consistency (linearizability) of read/write operations. PoWerStore's efficiency stems from combining lightweight cryptography, erasure coding and metadata write-backs, where readers write-back only metadata to achieve strong consistency. Central to PoWerStore is the concept of “Proofs of Writing” (PoW), a novel data storage technique inspired by commitment schemes. PoW rely on a 2-round write procedure, in which the first round writes the actual data and the second round only serves to “prove” the occurrence of the first round. PoW enable efficient implementations of strongly consistent BFT storage through metadata write-backs and low latency reads. We implemented PoWerStore and show its improved performance when compared to state of the art robust storage protocols, including protocols that tolerate only crash faults.

Failure detectors for crash faults in cloud

Failure detector (FD) is an inherent component in atomic broadcast and consensus protocols. Failures are broadly categorized into two types: crash and byzantine. The crash failures simply discontinue the working of a system whereas byzantine reflects the malicious behavior while ongoing communication. The problem to detect a failure becomes more challenging in a dynamic asynchronous environment like cloud computing. The paper proposes a failure detector in order to handle crash faults in cloud while addressing scalability. We introduce BCMP networks in order to compute performance parameters of the proposed algorithm, thereby, detecting failures in an accurate manner. Although, failure detection schemes have a tradeoff between efficiency and latency, the proposed algorithm achieves optimal balance between both metrics.

Group search of the plane with faulty robots

A collection of k mobile robots, initially placed at the origin of the plane, are searching for a stationary target. Each robot ri has a unit visibility range and can move no faster than its maximal speed vi, for i=1,2,...,k. The robots can communicate between themselves. We consider two communication models: wireless, in which a message sent by a robot can reach all other robots immediately, regardless of their positions, and face-to-face (F2F), in which robots can only exchange information when they are meeting. We assume that up to f robots, where f<k, may be unreliable. We consider two models of unreliability: (1) crash faults, in which we deal with an absence of some of the robots' capabilities (communication, perception, motion, etc.), and (2) byzantine faults, in which the robots may be malicious in that they may execute a wrong algorithm (e.g., transmitting wrong information).The goal is to minimize the group search time, which is equal to the time of arrival to the target of the last reliable robot. This is expressed as a function of d, the distance from the origin to the target. Our proposed algorithms for crash faults are asymptotically optimal in d in both communication models. For byzantine faults, our algorithm is asymptotically optimal for the wireless model. In the F2F model, we propose two algorithms: the first one has a competitive ratio of 2, while the second algorithm works for k≥2f+2 and is optimal when the robots speeds are all equal.