Checkpointing Protocol Research Articles

A checkpointing mechanism for virtual clusters using memory-bound time-multiplexed data transfers

Transparent hypervisor-level checkpoint-restart mechanisms for virtual clusters (VCs) or clusters of virtual machines (VMs) offer an attractive fault tolerance capability for cloud data centers. However, existing mechanisms have suffered from high checkpoint downtimes and overheads. This paper introduces Mekha, a novel hypervisor-level, in-memory coordinated checkpoint-restart mechanism for VCs that leverages precopy live migration. During a VC checkpoint event, Mekha creates a shadow VM for each VM and employs a novel memory-bound timed-multiplex data (MTD) transfer mechanism to replicate the state of each VM to its corresponding shadow VM. We also propose a global ending condition that enables the checkpoint coordinator to control the termination of the MTD algorithm for every VM in a VC, thereby reducing overall checkpoint latency. Furthermore, the checkpoint protocols of Mekha are designed based on barrier synchronizations and virtual time, ensuring the global consistency of checkpoints and utilizing existing data retransmission capabilities to handle message loss. We conducted several experiments to evaluate Mekha using a message passing interface (MPI) application from the NASA advanced supercomputing (NAS) parallel benchmark. The results demonstrate that Mekha significantly reduces checkpoint downtime compared to traditional checkpoint mechanisms. Consequently, Mekha effectively decreases checkpoint overheads while offering efficiency and practicality, making it a viable solution for cloud computing environments.

Scalable Communication-Induced Checkpointing Protocol with Little Overhead for Distributed Computing Environments

The existing communication-induced checkpointing protocols may not scale well due to their slow acquisition of the most recent timestamps of the next checkpoints of other processes. Accurate situation awareness with diversified information conveyance paths is needed to reduce the number of unnecessary forced checkpoints taken as few as possible. In this paper, a scalable communication-induced checkpointing protocol is proposed to considerably cut down the possibility of performing unnecessary forced checkpointing by exploiting the beneficial features of reliable communication channels. The protocol enables the sender of an application message to swiftly attain the most recent timestamp-related information of the next checkpoint of its receiver and accelerate the spread of the information to others, with little overhead. This behavioral feature may significantly elevate the accuracy of the awareness of the situations in which forced checkpointing is actually needed for useless checkpoint-free recovery. In addition, it generates no extra control message and no message logging overhead while significantly lessening the latency of message sending. Moreover, the protocol can always be operated under the non-deterministic execution model. The evaluation results indicate that the proposed protocol outperforms the existing ones at the reduced forced checkpointing overheads from 12.5% to 84.2%, and at the reduced total execution times from 2.5% to 11.5%.

Benchmarking and Configuring Security Levels in Intermittent Computing

Intermittent computing derives its name from the intermittent character of the power source used to drive the computing, typically an energy harvester of ambient energy sources. Intermittent computing is characterized by frequent transitions between the powered and the non-powered state. To enable the processor to quickly recover from unexpected power loss, regular checkpoints store the run-time state of the program, including variables, control information, and machine state. In sensitive applications such as logged measurements, checkpoints must be secured against tamper and replay. We investigate the overhead of creating, securing, and restoring checkpoints with respect to the application. We propose a configurable checkpoint security setting that leverages application properties to reduce overhead of checkpoint security and implement the same using a secure checkpointing protocol. We discuss a prototype implementation for a FRAM-based micro-controller, and we characterize the cost of adding and configuring security to traditional checkpointing using a suite of embedded benchmark applications.

Real-time Attack-recovery for Cyber-physical Systems Using Linear-quadratic Regulator

The increasing autonomy and connectivity in cyber-physical systems (CPS) come with new security vulnerabilities that are easily exploitable by malicious attackers to spoof a system to perform dangerous actions. While the vast majority of existing works focus on attack prevention and detection, the key question is “what to do after detecting an attack?”. This problem attracts fairly rare attention though its significance is emphasized by the need to mitigate or even eliminate attack impacts on a system. In this article, we study this attack response problem and propose novel real-time recovery for securing CPS. First, this work’s core component is a recovery control calculator using a Linear-Quadratic Regulator (LQR) with timing and safety constraints. This component can smoothly steer back a physical system under control to a target state set before a safe deadline and maintain the system state in the set once it is driven to it. We further propose an Alternating Direction Method of Multipliers (ADMM) based algorithm that can fast solve the LQR-based recovery problem. Second, supporting components for the attack recovery computation include a checkpointer, a state reconstructor, and a deadline estimator. To realize these components respectively, we propose (i) a sliding-window-based checkpointing protocol that governs sufficient trustworthy data, (ii) a state reconstruction approach that uses the checkpointed data to estimate the current system state, and (iii) a reachability-based approach to conservatively estimate a safe deadline. Finally, we implement our approach and demonstrate its effectiveness in dealing with totally 15 experimental scenarios which are designed based on 5 CPS simulators and 3 types of sensor attacks.

Effectual Non-Intrusive Minimum-Process Synchronous Checkpointing Protocol for Mobile Distributed Systems

While dealing with Mobile Distributed systems, we come across some issues like: mobility, low bandwidth of wireless channels and dearth of stable storage on mobile nodes, disconnections, inadequate battery power and high failure rate of mobile nodes. Minimum-process coordinated checkpointing is considered an attractive methodology to introduce fault tolerance in mobile systems transparently. In this paper, we propose a non-blocking coordinated global state compilation algorithm for mobile computing systems, which requires only a minimum number of processes to take permanent recovery points. We reduce the communication complexity as compared to the Cao-Singhal algorithm [4], while keeping the number of useless recovery points unchanged. Finally, the paper presents an optimization technique, which significantly reduces the number of useless recovery points at the cost of minor increase in the communication complexity. In coordinated global state compilation, if a single process fails to take its tentative recovery point; all the recovery point effort is aborted. We try to reduce this effort by taking soft recovery points in the first phase at Mobile Hosts.

Communication-Induced Checkpointing with Message Logging beyond the Piecewise Deterministic (PWD) Model for Distributed Systems

This paper introduces an effective communication-induced checkpointing protocol using message logging to enable the number of extra checkpoints to be far lower than the previous number. Even if a situation occurs in which it is decided that a process receiving a message has to perform forced checkpointing, our protocol allows the process to skip the forced checkpointing action if it recognizes that the state of its sender right before the receipt of the message is recoverable. Additionally, the communication-induced checkpointing protocol is thus not required to assume the piecewise deterministic model, despite being combined with message logging. This protocol can maintain these features by piggybacking a one-bit variable and an n-size vector on each message sent. Our simulation results verify our claim that the presented protocol performs much better than the representative optimized protocol with respect to the forced checkpointing frequency, regardless of the communication pattern.

Incentives in Ethereum's hybrid Casper protocol

SummaryWe present an overview of hybrid Casper the Friendly Finality Gadget (FFG), a proof‐of‐stake checkpointing protocol overlaid onto Ethereum's proof‐of‐work blockchain. We describe its core functionalities and reward scheme and explore its properties. Our findings indicate that Casper's implemented incentives mechanism ensures liveness, while providing safety guarantees that improve over standard proof‐of‐work protocols. On the basis of a minimal impact implementation of the protocol as a smart contract on the blockchain, we discuss additional issues related to parametrization, funding, throughput, and network overhead and detect potential limitations.

Efficient Communication Induced Checkpointing Protocol for Broadcast Network-based Distributed Systems

This paper proposes an enhanced Fully Informed Communication-Induced Checkpointing (FI-CIC) protocol to highly improve the possibility of detecting Z-cycle free patterns with no extra control message by utilizing the advantageous feature of the broadcast network in an effective way compared with the original FI-CIC protocol. Experimental results show that our protocol outperforms the previous one in terms of the number of forced checkpoints per process.

Study of Blockchains’s Consensus Mechanism Based on Credit

Practical Byzantine fault tolerance (PBFT) is one of the most popular consensus protocols of the blockchain. However, in the PBFT, the enthusiasm of reliable nodes cannot be stimulated effectively, and a large amount of communication resources are used for data consistency. Therefore, a new consensus protocol-credit-delegated Byzantine fault tolerance (CDBFT)-is proposed in this paper. The CDBFT works as the following: 1) a voting rewards and punishments scheme and its corresponding credit evaluation scheme are proposed not only to stimulate enthusiasm of reliable nodes but also to reduce the participation of abnormal nodes in the consensus process, and the virtuous circle of the system can be founded and 2) consistency and checkpoint protocols based on PBFT are proposed to improve the efficiency and flexibility of system. From the simulation results, a conclusion can be drawn, the participation probability of abnormal nodes in the consensus process can be reduced to 5%, and the efficiency and stability of the system are improved greatly in the long-time running.

Low overhead communication‐induced checkpointing protocols ensuring rollback‐dependency trackability property

SummaryCommunication‐induced checkpointing (CIC) protocols aim at finding consistent global checkpoint from which a system can safely restart without the risk of domino effect. CIC class gives processes the maximum autonomy for taking their local checkpoints while directs them to take forced checkpoints when harm patterns are detected. Such patterns can constitute of non‐causally doubled Z‐paths that violate the causal relation and can cause checkpoints uselessness, which results in hidden dependencies between checkpoints. Ensuring the absence of hidden dependencies hence means the ensure of the Rollback‐Dependency Trackability (RDT) property.Reducing the control information overhead is the main aim of our proposals. Our first proposal is called CSFDAS (Constant Size Fixed Dependency After Send), and the second is called CSRDTParner (Constant Size RDTPartner). Unlike to the previous works, the new proposed protocols use direct dependency clocks instead of transitive dependency vectors while keeping only a constant size of control information on messages. Conducted simulations show that the new protocols achieve a good performance compared to the RDT protocols proposed so far in the literature.

A novel hybrid protocol of checkpointing and rollback recovery for flat MANETs

With the advent of new technologies in the area of wireless systems and communications, modern distributed systems are subject to new constraints. Indeed, there is an increasing need to develop ad hoc distributed applications. However, the latter are now subject to new type of failures with more frequency. Therefore, fault tolerance is a crucial property to guarantee in this context and particularly for MANETs which are considered in this paper. We propose for this effect, a new hybrid checkpointing and rollback recovery protocols for flat MANETs, called two phases algorithm of checkpointing for adhoc networks (2PACA) networks. This protocol operates in the background of the distributed application and allows the latter to recover from a failure and to resume its processing. We advocate the use of our protocol in the context of flat MANETs and evaluate its performances by reporting some exhaustive simulation results performed on NS2 platform.

Towards Optimal Multi-Level Checkpointing

We provide a framework to analyze multi-level checkpointing protocols, by formally defining a $k$ -level checkpointing pattern. We provide a first-order approximation to the optimal checkpointing period, and show that the corresponding overhead is in the order of $\sum _{\ell =1}^{k}\sqrt{2\lambda _\ell C_\ell}$ , where $\lambda _\ell$ is the error rate at level $\ell$ , and $C_\ell$ the checkpointing cost at level $\ell$ . This nicely extends the classical Young/Daly formula on single-level checkpointing. Furthermore, we are able to fully characterize the shape of the optimal pattern (number and positions of checkpoints), and we provide a dynamic programming algorithm to determine the optimal subset of levels to be used. Finally, we perform simulations to check the accuracy of the theoretical study and to confirm the optimality of the subset of levels returned by the dynamic programming algorithm. The results nicely corroborate the theoretical study, and demonstrate the usefulness of multi-level checkpointing with the optimal subset of levels.

Toward an Optimal Online Checkpoint Solution under a Two-Level HPC Checkpoint Model

The traditional single-level checkpointing method suffers from significant overhead on large-scale platforms. Hence, multilevel checkpointing protocols have been studied extensively in recent years. The multilevel checkpoint approach allows different levels of checkpoints to be set (each with different checkpoint overheads and recovery abilities), in order to further improve the fault tolerance performance of extreme-scale HPC applications. How to optimize the checkpoint intervals for each level, however, is an extremely difficult problem. In this paper, we construct an easy-to-use two-level checkpoint model. Checkpoint level 1 deals with errors with low checkpoint/recovery overheads such as transient memory errors, while checkpoint level 2 deals with hardware crashes such as node failures. Compared with previous optimization work, our new optimal checkpoint solution offers two improvements: (1) it is an online solution without requiring knowledge of the job length in advance, and (2) it shows that periodic patterns are optimal and determines the best pattern. We evaluate the proposed solution and compare it with the most up-to-date related approaches on an extreme-scale simulation testbed constructed based on a real HPC application execution. Simulation results show that our proposed solution outperforms other optimized solutions and can improve the performance significantly in some cases. Specifically, with the new solution the wall-clock time can be reduced by up to 25.3 percent over that of other state-of-the-art approaches. Finally, a brute-force comparison with all possible patterns shows that our solution is always within 1 percent of the best pattern in the experiments.

Checkpointing with Minimal Recovery in Adhoc Net Based TMR

This paper describes two-fold approach towards utilizing Triple Modular Redundancy (TMR) in Wireless Adhoc Network (AdocNet). A distributed checkpointing and recovery protocol is proposed. The protocol eliminates useless checkpoints and helps in selecting only dependent processes in the concerned checkpointing interval, to recover. A process starts recovery from its last checkpoint only if it finds that it is dependent (directly or indirectly) on the faulty process. The recovery protocol also prevents the occurrence of missing or orphan messages. In AdocNet, a set of three nodes (connected to each other) is considered to form a TMR set, being designated as main, primary and secondary. A main node in one set may serve as primary or secondary in another. Computation is not triplicated, but checkpoint by main is duplicated in its primary so that primary can continue if main fails. Checkpoint by primary is then duplicated in secondary if primary fails too.

SAND: A fault-tolerant streaming architecture for network traffic analytics

Many long-running network analytics applications (e.g., flow size estimation and heavy traffic detection) impose a high-throughput and high reliability requirements on stream processing systems. However, previous stream processing systems which are designed for higher layer applications cannot sustain high-speed traffic at the core router level. Furthermore, due to the nondeterministic nature of message passing among workers, the fault-tolerant schemes of previous streaming architectures based on the continuous operator model cannot provide strong consistency which is essential for network analytics. In this paper, we present the design and implementation of SAND, a fault-tolerant distributed stream processing system for network analytics. SAND is designed to operate under high-speed network traffic, and it uses a novel checkpointing protocol which can perform failure recovery based on upstream backup and checkpointing. We prove our fault-tolerant scheme provides strong consistency even under multiple node failure. We implement several real-world network analytics applications on SAND, including heavy traffic hitter detection as well as policy and charging control for cellular networks, and we evaluate their performance using network traffic captured from commercial cellular core networks. We demonstrate that SAND can sustain high-speed network traffic and that our fault-tolerant scheme is efficient.

FNB: Fast Non-Blocking Coordinated Checkpointing Protocol for Distributed Systems

This paper presents a Fast Non-Blocking coordinated checkpointing protocol for distributed systems with the aim of minimizing the number of requests and mutable checkpoints while reducing the checkpointing latency. Our protocol relies on two mechanisms; the first one is piggybacking dependency information on computation and reply message, thereby, tracking direct, transitive and hidden dependencies among processes. The second one is popular processes; due to the communication between processes, it is more desirable that the checkpointing procedure is initiated by popular processes having more dependency information. In fact, this way may reduce the checkpointing latency and the likelihood of checkpointing halting caused by fault occurrence. We also present a simulation study that compares our protocol to CSNB protocol (Cao and Singhal Non-Blocking) and CSB.protocol (Cao and Singhal Blocking)

CDMCR: multi‐level fault‐tolerant system for distributed applications in cloud

AbstractCloud provides users with a new model of utilizing the computing infrastructure with the ability to perform parallel and distributed computations using elastic virtual cluster. However, the multi‐level and complex features make cloud computing system more prone to failure. In this paper, we present a multi‐level fault‐tolerant system for distributed applications in cloud named Distributed‐application oriented Multi‐level Checkpoint/Restart for Cloud (CDMCR). The CDMCR system backups the complete state of applications periodically with a snapshot‐based distributed checkpointing protocol, including file system state. Thus, we cannot only recover processes but also rollback data. A multi‐level recovery strategy is proposed, which includes process‐level recovery, virtual machine recreation, and host rescheduling, enabling comprehensive and efficient fault tolerance for different components in cloud. We deploy CDMCR as PaaS, so that users can be liberated from node management and system configuration and get access to fault‐tolerant service conveniently. We have implemented this system based on the Xen virtualization platform and the OpenNebula cloud platform. Experiments on the prototype demonstrate the correctness of the system. Analysis shows that CDMCR does not cause message loss or data loss, and the backup time remains nearly constant as the number of nodes increases on virtual cluster. Copyright © 2015 John Wiley & Sons, Ltd.

ОБЕСПЕЧЕНИЕ ОТКАЗОУСТОЙЧИВОСТИ ВЫСОКОПРОИЗВОДИТЕЛЬНЫХ ВЫЧИСЛЕНИЙ С ПОМОЩЬЮ ЛОКАЛЬНЫХ КОНТРОЛЬНЫХ ТОЧЕК

One of the main problems that occur in the area of high-performance computing is to continue computations despite of failures. In this paper, we consider the main definitions relating to dependability, briefly review the failure rates for distributed systems and also survey the rollback recovery approaches. The classic fault tolerance technique used in parallel applications is the coordinated checkpointing protocol. This protocol takes a consistent global checkpoint snapshot by capturing the local state of each process node simultaneously and saves it on a parallel file system via I/O nodes. However, as the number of compute nodes increases and the size of applications grow, the performance overhead of this protocol can reach an unacceptable level. A solution to this problem is to use local storage for checkpointing. To provide protection, it is necessary to duplicate checkpoints to other local storages. In this work, we develop user level approach and present scheme for checkpointing to the local storages. We proof that, if the number of failures is less than the maximum allowable value for the scheme then it is possible to recover from consistent global checkpoint.

CONSISTENCY OF DISTRIBUTED SYSTEM WITH ACTIVE INITIATOR PROCESS WITHOUT USELESS CHECKPOINTS

Checkpointing mechanism is the one of the best attractive approach for providing software fault tolerance in distributed message passing systems. This paper aims to implement a distributed checkpointing technique, which eliminates the drawbacks of the centralized approach like “domino effect”, “useless checkpoint” (checkpoints that do not contribute to global consistency), and “hidden and zigzag” dependencies. The proposed checkpointing protocol has a checkpoint initiator, but, coordination among the local checkpoints is done in a distributed fashion. This guaranty that no message would be lost in case of failure occurs, has been maintained in this work by exchange of information among the processes. However, there is no central checkpoint initiator, but each of the processes takes turn to act as an initiator. Processes take local checkpoints only after being notified by the initiator. The processes synchronize their activities of the current checkpointing interval before finally committing their checkpoints. Thus, the checkpointing pattern described in this paper takes only those checkpoints that will contribute to the consistent global snapshot thereby eliminating the number of useless checkpoints.

Minimum mutable checkpoint-based coordinated checkpointing protocol for mobile distributed systems

Coordinated checkpointing is an attractive approach to add fault tolerance to mobile distributed systems transparently against various predictable and unexpected faults. This approach avoids domino effect and require minimum stable storage but may require additional synchronisation messages and blocking. Coordinated checkpointing overhead can be minimised either by minimising processes to checkpoint or by making checkpointing process non-blocking. In literature it has been reported that there cannot be a minimum process non-blocking coordinated checkpointing protocol. Minimisation of processes to checkpoint for an initiation can be combined with non-blocking by taking some additional checkpoints that may be discarded on completion of the checkpoint protocol unnecessary checkpoints. Such checkpoints are called mutable if stored on local memory of mobile host temporarily. In this paper, we investigated this problem further and designed an efficient coordinated checkpointing protocol that is non-blocking, requires coordination of only minimum number of processes and reduces the overhead of unnecessary checkpoints significantly. Simulation studies show that our protocol reduces the number of unnecessary checkpoints almost to zero.