State Machine Replication Protocol Research Articles

Improved Fast-Response Consensus Algorithm Based on HotStuff.

Recent Byzantine Fault-Tolerant (BFT) State Machine Replication (SMR) protocols increasingly focus on scalability and security to meet the growing demand for Distributed Ledger Technology (DLT) applications across various domains. Current BFT consensus algorithms typically require a single leader node to receive and validate votes from the majority process and broadcast the results, a design challenging to scale in large systems. We propose a fast-response consensus algorithm based on improvements to HotStuff, aimed at enhancing transaction ordering speed and overall performance of distributed systems, even in the presence of faulty copies. The algorithm introduces an optimistic response assumption, employs a message aggregation tree to collect and validate votes, and uses a dynamically adjusted threshold mechanism to reduce communication delay and improve message delivery reliability. Additionally, a dynamic channel mechanism and an asynchronous leader multi-round mechanism are introduced to address multiple points of failure in the message aggregation tree structure, minimizing dependence on a single leader. This adaptation can be flexibly applied to real-world system conditions to improve performance and responsiveness. We conduct experimental evaluations to verify the algorithm's effectiveness and superiority. Compared to the traditional HotStuff algorithm, the improved algorithm demonstrates higher efficiency and faster response times in handling faulty copies and transaction ordering.

Exploring Scalability of BFT Blockchain Protocols through Network Simulations

Novel Byzantine fault-tolerant (BFT) state machine replication protocols improve scalability for their practical use in distributed ledger technology, where hundreds of replicas must reach consensus. Evaluating the performance of BFT protocol implementations requires careful evaluation. We propose a new methodology using scalable network simulations to predict BFT protocol performance. Our simulation architecture allows for the integration of existing BFT implementations without modification or re-implementation, offering a cost-effective alternative to large-scale cloud experiments. We validate our method by comparing simulation results with real-world cloud deployments, showing that simulations can accurately predict performance at larger scales when network limitations dominate. In our study, we applied this methodology to assess the performance of several “blockchain-generation” BFT protocols, including HotStuff, Kauri, Narwhal & Tusk, and Bullshark, under realistic network conditions (with constrained 25 Mbit/s bandwidth) and induced faults. Kauri emerges as the top performer, achieving 6742 operations per second (op/s) with 128 replicas, outperforming BullShark (2318 op/s) and Tusk (1952 op/s). HotStuff, using secp256k1 and BLS signatures, reaches 494 op/s and 707 op/s, respectively, demonstrating the efficiency of BLS-signature aggregation for saving bandwidth. This study demonstrates that state-of-the-art asynchronous BFT protocols can achieve competitive throughput in large-scale, real-world scenarios.

An optimisation for a two‐round good‐case latency protocol

AbstractByzantine broadcast is a fundamental primitive in distributed computing. A highly efficient Byzantine broadcast protocol, motivated by the real‐world performance of practical state machine replication protocols, is increasingly needed. This article focuses on the state‐of‐the‐art partially synchronous Byzantine broadcast protocol proposed by Abraham et al. (PODC’21), which achieves optimal good‐case latency of two rounds and optimal resilience of n ≥ 5f − 1 in this setting. Each step of the protocol is analysed, and then improved by cutting down the number of messages required to be collected and transmitted in the heaviest step of the protocol by about half, without adding any extra cost. This benefits from a new property, named “spread”, that we identify and extract from the original protocol. It helps us to eliminate non‐essential work in its view‐change procedure. The authors also show that no further reduction is possible without violating security. A prototype is implemented and the performances of improved and original protocols are evaluated in the same environment. The results show that our improvement can achieve about 50% lower communication cost and 40% shorter latency at a scale of 100 replicas. The latency gap becomes wider as the scale further increases.

DBFT: A Byzantine Fault Tolerance Protocol With Graceful Performance Degradation

Byzantine Fault Tolerant (BFT) state machine replication protocols are used to achieve agreement among replicated servers with arbitrary faults. Most existing BFT protocols perform well in fault-free cases, but usually suffer from serious performance degradation when faults occur. In this paper, we present DBFT, a BFT protocol that realizes graceful performance degradation in faulty cases. The major novelty of DBFT lies in the double-response mechanism, which lets replica nodes deterministically respond to clients twice: one is after the speculative execution phase and the other is after the commitment phase. The double-response mechanism ensures good performance in spite of inconsistency in speculative execution. Also, to further alleviates undetectable performance attacks by a smartly malicious primary, we change primary upon every outstanding request. Moreover, DBFT does not involve clients in critical consensus operations so as to reduce the load of clients. We prove the correctness properties, i.e., safety and liveness, of DBFT. We conduct extensive experiments and the results show that, DBFT outperforms similar BFT protocols obviously in normal cases.

Improved Byzantine fault tolerance with fast consensus

AbstractThis article presents an improved Byzantine fault tolerance algorithm to tolerate Byzantine faults rapidly by minimizing the load of the network with a minimum cost that provides a very simple methodology to design Byzantine fault tolerance (BFT) state machine replication protocol with the most favorable flexibility. Although, we have taken the initiative of fault tolerance from earlier works and conversely perform a different approach than them. Our protocol just requires replicas forffaults instead of in previous Byzantine protocols to mitigate the attacks by faulty primary and replicas. It also improves prior algorithms in terms of different parameters and requires fewer cryptographic techniques. Similar to previous BFT protocols, this protocol also guarantees correctness.

Lodestone: An Efficient Byzantine Fault-Tolerant Protocol in Consortium Blockchains

We present Lodestone, a chain-based Byzantine fault-tolerant (BFT) state machine replication (SMR) protocol under partial synchrony. Lodestone enables replicas to achieve consensus with two phases of voting and enjoys (1) optimistic responsiveness and (2) linear communication complexity on average. Similar to the state-of-the-art chain-based BFT protocols, Lodestone can be optimized with a pipelining idea elegantly. We implement pipelined Lodestone and deploy experiments to evaluate its performance. The evaluation results demonstrate that Lodestone has a lower latency than HotStuff under various workloads.

Solution] Matchmaker Paxos: A Reconfigurable Consensus Protocol

State machine replication protocols, like MultiPaxos and Raft, are at the heart of numerous distributed systems. To tol- erate machine failures, these protocols must replace failed machines with new machines, a process known as reconfigu- ration. Reconfiguration has become increasingly important over time as the need for frequent reconfiguration has grown. Despite this, reconfiguration has largely been neglected in the literature. In this paper, we present Matchmaker Paxos and Matchmaker MultiPaxos, a reconfigurable consensus and state machine replication protocol respectively. Our protocols can perform a reconfiguration with little to no impact on the latency or throughput of command processing; they can per- form a reconfiguration in a few milliseconds; and they present a framework that can be generalized to other replication pro- tocols in a way that previous reconfiguration techniques can not. We provide proofs of correctness for the protocols and optimizations, and present empirical results from an open source implementation showing that throughput and latency do not change significantly during a reconfiguration.

SoK: A Generalized Multi-Leader State Machine Replication Tutorial

MultiPaxos and Raft are the two most popular and widely deployed state machine replication protocols. There is a more sophisticated family of generalized multi-leader state machine replication protocols like EPaxos, Caesar, and Atlas that have better performance, but they are extremely complicated and hard to understand. Due to their complexity, they have seen little to no industry adoption, and academically there has been a lack of clarity in analyzing, comparing, and extending the protocols. This paper is a tutorial on generalized multi-leader protocols. We explain why the protocols work the way they do, what they have in common, where they differ, which parts of the protocols are straightforward, which are more subtle than they appear, and so on. In doing so, we present four new generalized multi-leader protocols, identify key insights into existing protocols, and taxonomize the space.

Scaling replicated state machines with compartmentalization

State machine replication protocols, like MultiPaxos and Raft, are a critical component of many distributed systems and databases. However, these protocols offer relatively low throughput due to several bottlenecked components. Numerous existing protocols fix different bottlenecks in isolation but fall short of a complete solution. When you fix one bottleneck, another arises. In this paper, we introduce compartmentalization, the first comprehensive technique to eliminate state machine replication bottlenecks. Compartmentalization involves decoupling individual bottlenecks into distinct components and scaling these components independently. Compartmentalization has two key strengths. First, compartmentalization leads to strong performance. In this paper, we demonstrate how to compartmentalize MultiPaxos to increase its throughput by 6× on a write-only workload and 16× on a mixed read-write workload. Unlike other approaches, we achieve this performance without the need for specialized hardware. Second, compartmentalization is a technique, not a protocol. Industry practitioners can apply compartmentalization to their protocols incrementally without having to adopt a completely new protocol.

A fair selection protocol for committee-based permissionless blockchains

Committee-based blockchain consensus protocols combine permissionless consensus and classical state machine replication protocols to process transactions efficiently. Due to corruptions by the adversary, reconfiguration mechanisms have to be deployed to update committee members. How to select enough fraction of honest nodes is a key issue that needs to be addressed.In this paper, we propose a fair selection protocol for reconfiguring a committee in a permissionless blockchain. Our fair selection protocol consists of two main phases: the mining process and the confirmation of the new nodes list. We analyze the impact of the network latency and give a rigorous proof of the mining process. Furthermore, we point out that in the process of confirming the new nodes list, “node censorship” by a malicious leader could lead to an increase in the proportion of new nodes controlled by an adversary. We propose a threshold-vote rule to defend against the node censorship attack and prove the security of the entire protocol, including safety and liveness properties. Our fair selection protocol could be implemented in committee-based permissionless blockchains according to different demands with proper parameters.

Leader Set Selection for Low-Latency Geo-Replicated State Machine

Modern planetary scale distributed systems largely rely on a State Machine Replication protocol to keep their service reliable, yet it comes with a specific challenge: latency, bounded by the speed of light. In particular, clients of a single-leader protocol, such as Paxos, must communicate with the leader which must in turn communicate with other replicas: inappropriate selection of a leader may result in unnecessary round-trips across the globe. To cope with this limitation, several all-leader and leaderless alternatives have been proposed recently. Unfortunately, none of them fits all circumstances. In this article we argue that the “right” choice of the number of leaders depends on a given replica configuration and the workload. Then we present ${\mathsf {Droopy}}$ and ${\mathsf {Dripple}}$ , two sister approaches built upon state machine replication protocols. ${\mathsf {Droopy}}$ dynamically reconfigures the set of leaders. Whereas, ${\mathsf {Dripple}}$ coordinates state partitions wisely, so that each partition can be reconfigured (by ${\mathsf {Droopy}}$ ) separately. Our experimental evaluation on Amazon EC2 shows that, ${\mathsf {Droopy}}$ and ${\mathsf {Dripple}}$ reduce latency under imbalanced or localized workloads, compared to their native protocol. When most requests are non-commutative, our approaches do not affect the performance of their native protocol and both outperform a state-of-the-art leaderless protocol.

How many planet-wide leaders should there be?

Geo-replication becomes increasingly important for modern planetary scale distributed systems, yet it comes with a specific challenge: latency, bounded by the speed of light. In particular, clients of a geo-replicated system must communicate with a leader which must in turn communicate with other replicas: wrong selection of a leader may result in unnecessary round-trips across the globe. Classical protocols such as celebrated Paxos, have a single leader making them unsuitable for serving widely dispersed clients. To address this issue, several all-leader geo-replication protocols have been proposed recently, in which every replica acts as a leader. However, because these protocols require coordination among all replicas, commiting a client's request at some replica may incure the so-called "delayed commit" problem, which can introduce even a higher latency than a classical single-leader majority-based protocol such as Paxos. In this paper, we argue that the "right" choice of the number of leaders in a geo-replication protocol depends on a given replica configuration and propose Droopy, an optimization for state machine replication protocols that explores the space between single-leader and all-leader by dynamically reconfiguring the leader set. We implement Droopy on top of Clock-RSM, a state-of-the-art all-leader protocol. Our evaluation on Amazon EC2 shows that, under typical imbalanced workloads, Droopy-enabled Clock-RSM efficiently reduces latency compared to native Clock-RSM, whereas in other cases the latency is the same as that of the native Clock-RSM.

The Next 700 BFT Protocols

We present Abstract (ABortable STate mAChine replicaTion), a new abstraction for designing and reconfiguring generalized replicated state machines that are, unlike traditional state machines, allowed to abort executing a client’s request if “something goes wrong.” Abstract can be used to considerably simplify the incremental development of efficient Byzantine fault-tolerant state machine replication ( BFT ) protocols that are notorious for being difficult to develop. In short, we treat a BFT protocol as a composition of Abstract instances. Each instance is developed and analyzed independently and optimized for specific system conditions. We illustrate the power of Abstract through several interesting examples. We first show how Abstract can yield benefits of a state-of-the-art BFT protocol in a less painful and error-prone manner. Namely, we develop AZyzzyva , a new protocol that mimics the celebrated best-case behavior of Zyzzyva using less than 35% of the Zyzzyva code. To cover worst-case situations, our abstraction enables one to use in AZyzzyva any existing BFT protocol. We then present Aliph , a new BFT protocol that outperforms previous BFT protocols in terms of both latency (by up to 360%) and throughput (by up to 30%). Finally, we present R-Aliph , an implementation of Aliph that is robust , that is, whose performance degrades gracefully in the presence of Byzantine replicas and Byzantine clients.

HT-Paxos: high throughput state-machine replication protocol for large clustered data centers.

Paxos is a prominent theory of state-machine replication. Recent data intensive systems that implement state-machine replication generally require high throughput. Earlier versions of Paxos as few of them are classical Paxos, fast Paxos, and generalized Paxos have a major focus on fault tolerance and latency but lacking in terms of throughput and scalability. A major reason for this is the heavyweight leader. Through offloading the leader, we can further increase throughput of the system. Ring Paxos, Multiring Paxos, and S-Paxos are few prominent attempts in this direction for clustered data centers. In this paper, we are proposing HT-Paxos, a variant of Paxos that is the best suitable for any large clustered data center. HT-Paxos further offloads the leader very significantly and hence increases the throughput and scalability of the system, while at the same time, among high throughput state-machine replication protocols, it provides reasonably low latency and response time.

A Method of Parallelizing Consensuses for Accelerating Byzantine Fault Tolerance

We propose a new method that accelerates asynchronous Byzantine Fault Tolerant (BFT) protocols designed on the principle of state machine replication. State machine replication protocols ensure consistency among replicas by applying operations in the same order to all of them. A naive way to determine the application order of the operations is to repeatedly execute the BFT consensus to determine the next executed operation, but this may introduce inefficiency caused by waiting for the completion of the previous execution of the consensus protocol. To reduce this inefficiency, our method allows parallel execution of the consensuses while keeping consistency of the consensus results at the replicas. In this paper, we also prove the correctness of our method and experimentally compare it with the existing method in terms of latency and throughput. The evaluation results show that our method makes a BFT protocol three or four times faster than the existing one when some machines or message transmissions are delayed.

Optimizing Paxos with batching and pipelining

Paxos is probably the most popular state machine replication protocol. Two optimizations that can greatly improve its performance are batching and pipelining. However, tuning these two optimizations to achieve high-throughput can be challenging, as their effectiveness depends on many parameters like the network latency and bandwidth, the speed of the nodes, and the properties of the application. We address this question by presenting an analytical model of the performance of Paxos, and showing how it can be used to determine configuration values for the batching and pipelining optimizations that result in high-throughput. We then validate the model, by presenting the results of experiments where we investigate the interaction of these two optimizations both in LAN and WAN environments, and comparing these results with the prediction from the model. The results confirm the accuracy of the model, with the predicted values being usually very close to the ones that provide the highest performance in the experiments. Furthermore, both the model and the experiments give useful insights into the relative effectiveness of batching and pipelining. They show that although batching by itself provides the largest gains in all scenarios, in most cases combining it with pipelining provides a very significant additional increase in throughput, with this being true not only on high-latency WAN but also in many LAN scenarios.