Fault-tolerant State Machine Replication Research Articles

Exploring Scalability of BFT Blockchain Protocols through Network Simulations

Novel Byzantine fault-tolerant (BFT) state machine replication protocols improve scalability for their practical use in distributed ledger technology, where hundreds of replicas must reach consensus. Evaluating the performance of BFT protocol implementations requires careful evaluation. We propose a new methodology using scalable network simulations to predict BFT protocol performance. Our simulation architecture allows for the integration of existing BFT implementations without modification or re-implementation, offering a cost-effective alternative to large-scale cloud experiments. We validate our method by comparing simulation results with real-world cloud deployments, showing that simulations can accurately predict performance at larger scales when network limitations dominate. In our study, we applied this methodology to assess the performance of several “blockchain-generation” BFT protocols, including HotStuff, Kauri, Narwhal & Tusk, and Bullshark, under realistic network conditions (with constrained 25 Mbit/s bandwidth) and induced faults. Kauri emerges as the top performer, achieving 6742 operations per second (op/s) with 128 replicas, outperforming BullShark (2318 op/s) and Tusk (1952 op/s). HotStuff, using secp256k1 and BLS signatures, reaches 494 op/s and 707 op/s, respectively, demonstrating the efficiency of BLS-signature aggregation for saving bandwidth. This study demonstrates that state-of-the-art asynchronous BFT protocols can achieve competitive throughput in large-scale, real-world scenarios.

Improved Fast-Response Consensus Algorithm Based on HotStuff.

Recent Byzantine Fault-Tolerant (BFT) State Machine Replication (SMR) protocols increasingly focus on scalability and security to meet the growing demand for Distributed Ledger Technology (DLT) applications across various domains. Current BFT consensus algorithms typically require a single leader node to receive and validate votes from the majority process and broadcast the results, a design challenging to scale in large systems. We propose a fast-response consensus algorithm based on improvements to HotStuff, aimed at enhancing transaction ordering speed and overall performance of distributed systems, even in the presence of faulty copies. The algorithm introduces an optimistic response assumption, employs a message aggregation tree to collect and validate votes, and uses a dynamically adjusted threshold mechanism to reduce communication delay and improve message delivery reliability. Additionally, a dynamic channel mechanism and an asynchronous leader multi-round mechanism are introduced to address multiple points of failure in the message aggregation tree structure, minimizing dependence on a single leader. This adaptation can be flexibly applied to real-world system conditions to improve performance and responsiveness. We conduct experimental evaluations to verify the algorithm's effectiveness and superiority. Compared to the traditional HotStuff algorithm, the improved algorithm demonstrates higher efficiency and faster response times in handling faulty copies and transaction ordering.

A Survey of Consortium Blockchain and Its Applications

Blockchain is a revolutionary technology that has reshaped the trust model among mutually distrustful peers in a distributed network. While blockchain is well-known for its initial usage in a public manner, such as the cryptocurrency of Bitcoin, consortium blockchain, which requires authentication of all involved participants, has also been widely adopted in various domains. Nevertheless, there is a lack of comprehensive study of consortium blockchain in terms of its architecture design, consensus mechanisms, comparative performance, etc. In this study, we aim to fill this gap by surveying the most popular consortium blockchain platforms and assessing their core designs in a layered fashion. Particularly, Byzantine fault tolerant (BFT) state machine replication (SMR) is introduced to act as a basic computational model of consortium blockchain. Then the consortium blockchain is split into the hardware layer, layer-0 (network layer), layer-I (data layer, consensus layer and contract layer), layer-II protocols, and application layer. Each layer is presented with closely related discussion and analysis. Furthermore, with the extraction of the core functionalities, i.e., robust storage and guaranteed execution, that a consortium blockchain can provide, several typical consortium blockchain-empowered decentralized application scenarios are introduced. With these thorough studies and analyses, this work aims to systematize the knowledge dispersed in the consortium blockchain, highlight the unsolved challenges, and also indicate the propitious avenues of future work.

Leaderless consensus

Classic synchronous consensus algorithms are leaderless in that processes exchange proposals, choose the maximum value and decide when they see the same choice across a couple of rounds. Indulgent consensus algorithms are typically leader-based. Although they tolerate unexpected delays and find practical applications in blockchains running over an open network like the Internet, their performance is highly dependent on the activity of a single participant. This paper asks whether, under eventual synchrony, it is possible to deterministically solve consensus without a leader. The fact that the weakest failure detector to solve consensus is one that also eventually elects a leader seems to indicate that the answer to the question is negative. We prove in this paper that the answer is actually positive. We first give a precise definition of the very notion of a leaderless algorithm. Then we present three indulgent leaderless consensus algorithms, each we believe interesting in its own right: (i) for shared memory, (ii) for message passing with omission failures and (iii) for message passing with Byzantine failures. Finally, we implement a Byzantine fault tolerant (BFT) state machine replication (SMR), that is leaderless. Our empirical results demonstrate that it is faster and more robust than HotStuff, the recent BFT SMR algorithm used in the Facebook Libra blockchain when deployed in a wide area network.

Deterministic or probabilistic? - A survey on Byzantine fault tolerant state machine replication

Byzantine Fault tolerant (BFT) protocols are implemented to guarantee the correct system/application behavior even in the presence of arbitrary faults (i.e., Byzantine faults). Byzantine Fault tolerant State Machine Replication (BFT-SMR) is a known software solution for masking arbitrary faults and malicious attacks (Liu et al., 2020). In this survey, we present and discuss relevant BFT-SMR protocols, focusing on deterministic and probabilistic approaches. The main purpose of this paper is to discuss the characteristics of proposed works for each approach, as well as identify the trade-offs for each different approach.

DBFT: A Byzantine Fault Tolerance Protocol With Graceful Performance Degradation

Byzantine Fault Tolerant (BFT) state machine replication protocols are used to achieve agreement among replicated servers with arbitrary faults. Most existing BFT protocols perform well in fault-free cases, but usually suffer from serious performance degradation when faults occur. In this paper, we present DBFT, a BFT protocol that realizes graceful performance degradation in faulty cases. The major novelty of DBFT lies in the double-response mechanism, which lets replica nodes deterministically respond to clients twice: one is after the speculative execution phase and the other is after the commitment phase. The double-response mechanism ensures good performance in spite of inconsistency in speculative execution. Also, to further alleviates undetectable performance attacks by a smartly malicious primary, we change primary upon every outstanding request. Moreover, DBFT does not involve clients in critical consensus operations so as to reduce the load of clients. We prove the correctness properties, i.e., safety and liveness, of DBFT. We conduct extensive experiments and the results show that, DBFT outperforms similar BFT protocols obviously in normal cases.

Lodestone: An Efficient Byzantine Fault-Tolerant Protocol in Consortium Blockchains

We present Lodestone, a chain-based Byzantine fault-tolerant (BFT) state machine replication (SMR) protocol under partial synchrony. Lodestone enables replicas to achieve consensus with two phases of voting and enjoys (1) optimistic responsiveness and (2) linear communication complexity on average. Similar to the state-of-the-art chain-based BFT protocols, Lodestone can be optimized with a pipelining idea elegantly. We implement pipelined Lodestone and deploy experiments to evaluate its performance. The evaluation results demonstrate that Lodestone has a lower latency than HotStuff under various workloads.

Byzantine Fault-tolerant State-machine Replication from a Systems Perspective

Byzantine fault-tolerant (BFT) state-machine replication makes it possible to design systems that are resilient against arbitrary faults, a requirement considered crucial for an increasing number of use cases such as permissioned blockchains, firewalls, and SCADA systems. Unfortunately, the strong fault-tolerance guarantees provided by BFT replication protocols come at the cost of a high complexity, which is why it is inherently difficult to correctly implement BFT systems in practice. This is all the more true with regard to the plethora of solutions and ideas that have been developed in recent years to improve performance, availability, or resource efficiency. This survey aims at facilitating the task of building BFT systems by presenting an overview of state-of-the-art techniques and analyzing their practical implications, for example, with respect to applicability and composability. In particular, this includes problems that arise in the context of concrete implementations, but which are often times passed over in literature. Starting with an in-depth discussion of the most important architectural building blocks of a BFT system (i.e., clients, agreement protocol, execution stage), the survey then focuses on selected approaches and mechanisms addressing specific tasks such as checkpointing and recovery.

Asphalion: trustworthy shielding against Byzantine faults

Byzantine fault-tolerant state-machine replication (BFT-SMR) is a technique for hardening systems to tolerate arbitrary faults. Although robust, BFT-SMR protocols are very costly in terms of the number of required replicas (3f+1 to tolerate f faults) and of exchanged messages. However, with "hybrid" architectures, where "normal" components trust some "special" components to provide properties in a trustworthy manner, the cost of using BFT can be dramatically reduced. Unfortunately, even though such hybridization techniques decrease the message/time/space complexity of BFT protocols, they also increase their structural complexity. Therefore, we introduce Asphalion, the first theorem prover-based framework for verifying implementations of hybrid systems and protocols. It relies on three novel languages: (1) HyLoE: a Hybrid Logic of Events to reason about hybrid fault models; (2) MoC: a Monadic Component language to implement systems as collections of interacting hybrid components; and (3) LoCK: a sound Logic of events-based Calculus of Knowledge to reason about both homogeneous and hybrid systems at a high-level of abstraction (thereby allowing reusing proofs, and capturing the high-level logic of distributed systems). In addition, Asphalion supports compositional reasoning, e.g., through mechanisms to lift properties about trusted-trustworthy components, to the level of the distributed systems they are integrated in. As a case study, we have verified crucial safety properties (e.g., agreement) of several implementations of hybrid protocols.

Scalable Byzantine fault-tolerant state-machine replication on heterogeneous servers

When provided with more powerful or extra hardware, state-of-the-art Byzantine fault-tolerant (BFT) replication protocols are unable to effectively exploit the additional computing resources: on the one hand, in settings with heterogeneous servers existing protocols cannot fully utilize servers with higher performance capabilities. On the other hand, using more servers than the minimum number of replicas required for Byzantine fault tolerance in general does not lead to improved throughput and latency, but instead actually degrades performance. In this paper, we address these problems with Omada, a BFT system architecture that is able to benefit from additional hardware resources. To achieve this property while still providing strong consistency, Omada first parallelizes agreement into multiple groups and then executes the requests handled by different groups in a deterministic order. By varying the number of requests to be ordered between groups as well as the number of groups that a replica participates in between servers, Omada offers the possibility to individually adjust the resource usage per server. Moreover, the fact that not all replicas need to take part in every group enables the architecture to exploit additional servers.

The Next 700 BFT Protocols

We present Abstract (ABortable STate mAChine replicaTion), a new abstraction for designing and reconfiguring generalized replicated state machines that are, unlike traditional state machines, allowed to abort executing a client’s request if “something goes wrong.” Abstract can be used to considerably simplify the incremental development of efficient Byzantine fault-tolerant state machine replication ( BFT ) protocols that are notorious for being difficult to develop. In short, we treat a BFT protocol as a composition of Abstract instances. Each instance is developed and analyzed independently and optimized for specific system conditions. We illustrate the power of Abstract through several interesting examples. We first show how Abstract can yield benefits of a state-of-the-art BFT protocol in a less painful and error-prone manner. Namely, we develop AZyzzyva , a new protocol that mimics the celebrated best-case behavior of Zyzzyva using less than 35% of the Zyzzyva code. To cover worst-case situations, our abstraction enables one to use in AZyzzyva any existing BFT protocol. We then present Aliph , a new BFT protocol that outperforms previous BFT protocols in terms of both latency (by up to 360%) and throughput (by up to 30%). Finally, we present R-Aliph , an implementation of Aliph that is robust , that is, whose performance degrades gracefully in the presence of Byzantine replicas and Byzantine clients.

Byzantine fault tolerance for session-oriented multi-tiered applications

This article presents a lightweight Byzantine fault tolerance (BFT) framework for session-oriented multi-tiered applications. We conclude that it is sufficient to use a lightweight BFT algorithm instead of a traditional BFT algorithm, based on a comprehensive study of the threat model to, and the state model of, the session-oriented multi-tiered applications. The lightweight BFT algorithm uses source ordering, rather than total ordering, of incoming requests to achieve Byzantine fault tolerant state-machine replication of such type of applications. The performance of the lightweight BFT framework is evaluated using a shopping cart application prototype built on the web services platform. The same shopping cart application is used as a running example to illustrate the problem we address and our proposed solution. Performance evaluation results obtained from the prototype implementation show that indeed our lightweight BFT algorithm incurs very insignificant overhead.

Efficient Byzantine Fault-Tolerance

We present two asynchronous Byzantine fault-tolerant state machine replication (BFT) algorithms, which improve previous algorithms in terms of several metrics. First, they require only 2f+1 replicas, instead of the usual 3f+1. Second, the trusted service in which this reduction of replicas is based is quite simple, making a verified implementation straightforward (and even feasible using commercial trusted hardware). Third, in nice executions the two algorithms run in the minimum number of communication steps for nonspeculative and speculative algorithms, respectively, four and three steps. Besides the obvious benefits in terms of cost, resilience and management complexity-fewer replicas to tolerate a certain number of faults-our algorithms are simpler than previous ones, being closer to crash fault-tolerant replication algorithms. The performance evaluation shows that, even with the trusted component access overhead, they can have better throughput than Castro and Liskov's PBFT, and better latency in networks with nonnegligible communication delays.

Zyzzyva

A longstanding vision in distributed systems is to build reliable systems from unreliable components. An enticing formulation of this vision is Byzantine Fault-Tolerant (BFT) state machine replication, in which a group of servers collectively act as a correct server even if some of the servers misbehave or malfunction in arbitrary (“Byzantine”) ways. Despite this promise, practitioners hesitate to deploy BFT systems, at least partly because of the perception that BFT must impose high overheads. In this article, we present Zyzzyva, a protocol that uses speculation to reduce the cost of BFT replication. In Zyzzyva, replicas reply to a client's request without first running an expensive three-phase commit protocol to agree on the order to process requests. Instead, they optimistically adopt the order proposed by a primary server, process the request, and reply immediately to the client. If the primary is faulty, replicas can become temporarily inconsistent with one another, but clients detect inconsistencies, help correct replicas converge on a single total ordering of requests, and only rely on responses that are consistent with this total order. This approach allows Zyzzyva to reduce replication overheads to near their theoretical minima and to achieve throughputs of tens of thousands of requests per second, making BFT replication practical for a broad range of demanding services.

Zyzzyva

A longstanding vision in distributed systems is to build reliable systems from unreliable components. An enticing formulation of this vision is Byzantine fault-tolerant (BFT) state machine replication, in which a group of servers collectively act as a correct server even if some of the servers misbehave or malfunction in arbitrary (Byzantine) ways. Despite this promise, practitioners hesitate to deploy BFT systems at least partly because of the perception that BFT must impose high overheads. In this article, we present Zyzzyva, a protocol that uses speculation to reduce the cost of BFT replication. In Zyzzyva, replicas reply to a client's request without first running an expensive three-phase commit protocol to agree on the order to process requests. Instead, they optimistically adopt the order proposed by a primary server, process the request, and reply immediately to the client. If the primary is faulty, replicas can become temporarily inconsistent with one another, but clients detect inconsistencies, help correct replicas converge on a single total ordering of requests, and only rely on responses that are consistent with this total order. This approach allows Zyzzyva to reduce replication overheads to near their theoretical minima and to achieve throughputs of tens of thousands of requests per second, making BFT replication practical for a broad range of demanding services.

Zyzzyva

We present Zyzzyva, a protocol that uses speculation to reduce the cost and simplify the design of Byzantine fault tolerant state machine replication. In Zyzzyva, replicas respond to a client's request without first running an expensive three-phase commit protocol to reach agreement on the order in which the request must be processed. Instead, they optimistically adopt the order proposed by the primary and respond immediately to the client. Replicas can thus become temporarily inconsistent with one another, but clients detect inconsistencies, help correct replicas converge on a single total ordering of requests, and only rely on responses that are consistent with this total order. This approach allows Zyzzyva to reduce replication overheads to near their theoretical minimal.

Separating agreement from execution for byzantine fault tolerant services

We describe a new architecture for Byzantine fault tolerant state machine replication that separates agreement that orders requests from execution that processes requests. This separation yields two fundamental and practically significant advantages over previous architectures. First, it reduces replication costs because the new architecture can tolerate faults in up to half of the state machine replicas that execute requests. Previous systems can tolerate faults in at most a third of the combined agreement/state machine replicas. Second, separating agreement from execution allows a general privacy firewall architecture to protect confidentiality through replication. In contrast, replication in previous systems hurts confidentiality because exploiting the weakest replica can be sufficient to compromise the system. We have constructed a prototype and evaluated it running both microbenchmarks and an NFS server. Overall, we find that the architecture adds modest latencies to unreplicated systems and that its performance is competitive with existing Byzantine fault tolerant systems.