Uncoordinated Checkpointing Research Articles

Providing Consistent State to Distributed Storage System

In cloud storage systems, users must be able to shut down the application when not in use and restart it from the last consistent state when required. BlobSeer is a data storage application, specially designed for distributed systems, that was built as an alternative solution for the existing popular open-source storage system-Hadoop Distributed File System (HDFS). In a cloud model, all the components need to stop and restart from a consistent state when the user requires it. One of the limitations of BlobSeer DFS is the possibility of data loss when the system restarts. As such, it is important to provide a consistent start and stop state to BlobSeer components when used in a Cloud environment to prevent any data loss. In this paper, we investigate the possibility of BlobSeer providing a consistent state distributed data storage system with the integration of checkpointing restart functionality. To demonstrate the availability of a consistent state, we set up a cluster with multiple machines and deploy BlobSeer entities with checkpointing functionality on various machines. We consider uncoordinated checkpoint algorithms for their associated benefits over other alternatives while integrating the functionality to various BlobSeer components such as the Version Manager (VM) and the Data Provider. The experimental results show that with the integration of the checkpointing functionality, a consistent state can be ensured for a distributed storage system even when the system restarts, preventing any possible data loss after the system has encountered various system errors and failures.

Checkpointing Algorithms for Fault-Tolerant Execution of Large-Scale Distributed Applications in Cloud

Cloud computing provides infinite resources and a suitable environment for the execution of large scale computing applications. However, it is also susceptible to frequent failures which can affect users as well as service providers adversely. Therefore, fault tolerance techniques are necessary for the reliable execution of applications in the cloud. This work presents checkpointing based fault tolerance protocols for two types of distributed applications. The first kind of applications is the Bags of Tasks (BoT) applications where an application comprises of a set of independent tasks that do not communicate with each other during execution. Hence, an uncoordinated checkpointing algorithm is proposed for fault tolerance of BoT applications. Subsequently, we consider large scale distributed applications composed of multiple tasks dependent on each other due to inter-task message passing. An uncoordinated checkpointing and message logging protocol is presented for this type of applications. The proposed protocols utilize storage at edge switches in a data center to reduce the bandwidth consumption for saving checkpoints and message logs. Simulation results have demonstrated that the proposed protocols provide an increased rate of successful recoveries from failures and cause lower resource overhead than other contemporary and related schemes.

A Comparison of Application-Level Fault Tolerance Schemes for Task Pools

Fault tolerance is an important requirement for successful program execution on exascale systems. The common approach, checkpointing, regularly saves a program’s state, such that the execution can be restarted after permanent node failures. Checkpointing is often performed on system level, but its deployment on application level can reduce the running time overhead.The drawback of application-level checkpointing is a higher programming expense. It pays off if the checkpointing is applied to reusable patterns. We consider task pools, which exist in many variants. The paper supposes that tasks are generated dynamically and are free of side effects. Further, the final result must be computed from individual task results by reduction. Moreover, the pools must be distributed with private queues, and adopt work stealing.The paper describes and evaluates three application-level fault tolerance schemes for task pools. All use uncoordinated checkpointing and regularly save information in a resilient store. The first scheme (called AllFT) saves descriptors of all open tasks; the second scheme (called IncFT) selectively and incrementally saves only part of them; and the third scheme (called LogFT) logs stealing events and writes checkpoints in parallel to task processing.All schemes have been implemented by extending the Global Load Balancing (GLB) library of the “APGAS for Java” programming system. In experiments with the UTS, NQueens, and BC benchmarks with up to 672 workers, the running time overhead during failure-free execution, compared to a non-resilient version of GLB, was typically below 6%. The recovery cost was negligible, and there was no clear winner among the three schemes. A more detailed performance analysis with synthetic benchmarks revealed that IncFT and LogFT are superior in scenarios with large task descriptors.

The unexpected virtue of almost: Exploiting MPI collective operations to approximately coordinate checkpoints

SummaryCoordinated checkpoint/restart is currently the dominant approach to mitigating the impact of failures on important scientific applications running on large‐scale distributed systems. However, there is widespread evidence that coordinated checkpointing may no longer be viable on next‐generation systems. Uncoordinated checkpoint/restart attempts to address the shortcomings of coordinated checkpoint/restart by allowing application processes to checkpoint their state independently. However, eliminating coordination may significantly degrade application performance. In this paper, we propose an approach that leverages existing coordination in important scientific applications to approximately coordinate checkpoints. Specifically, we propose to extend MPI implementations to force checkpoints to occur immediately after the completion of a collective operation. We evaluate the performance implications of this approach using an existing validated simulation framework. Our results demonstrate that approximately coordinated checkpointing can significantly improve application performance relative to totally uncoordinated checkpointing. We also show that forcing checkpoints to occur following a collective operation has a small impact on the nominal checkpoint interval for several important workloads. As a whole, the results presented in this paper demonstrate that approximately coordinated checkpointing may provide significant performance benefits without significantly increasing the cost of failure recovery.

Hybrid Message Pessimistic Logging. Improving current pessimistic message logging protocols

With the growing scale of HPC applications, there has been an increase in the number of interruptions as a consequence of hardware failures. The remarkable decrease of Mean Time Between Failures (MTBF) in current systems encourages the research of suitable fault tolerance solutions. Message logging combined with uncoordinated checkpoint compose a scalable rollback-recovery solution. However, message logging techniques are usually responsible for most of the overhead during failure-free executions. Taking this into consideration, this paper proposes the Hybrid Message Pessimistic Logging (HMPL) which focuses on combining the fast recovery feature of pessimistic receiver-based message logging with the low failure-free overhead introduced by pessimistic sender-based message logging. The HMPL manages messages using a distributed controller and storage to avoid harming system’s scalability. Experiments show that the HMPL is able to reduce overhead by 34% during failure-free executions and 20% in faulty executions when compared with a pessimistic receiver-based message logging.

Message Efficient Checkpointing and Rollback Recovery in Heterogeneous Mobile Networks

Heterogeneous networks provide an appealing way of expanding the computing capability of mobile networks by combining infrastructure-less mobile ad-hoc networks with the infrastructure-based cellular mobile networks. The nodes in such a network range from low-power nodes to macro base stations and thus, vary greatly in their capabilities such as computation power and battery power. The nodes are susceptible to different types of transient and permanent failures and therefore, the algorithms designed for such networks need to be fault-tolerant. The article presents a checkpointing algorithm for the rollback recovery of mobile hosts in a heterogeneous mobile network. Checkpointing is a well established approach to provide fault tolerance in static and cellular mobile distributed systems. However, the use of checkpointing for fault tolerance in a heterogeneous environment remains to be explored. The proposed protocol is based on the results of zigzag paths and zigzag cycles by Netzer–Xu. Considering the heterogeneity prevalent in the network, an uncoordinated checkpointing technique is employed. Yet, useless checkpoints are avoided without causing a high message overhead.

A Trust-based Uncoordinated Checkpointing Algorithm in Mobile Ad Hoc Networks (MANETs)

The paper presents a trust-value based uncoordinated check pointing algorithm in Mobile Ad Hoc Networks (MANETs). We aim to improve the overall check pointing overhead incurred in the execution of recovery protocols. Most check pointing algorithms do not consider the mobility rate of nodes while taking a periodic checkpoint, thereby, resulting in poor utilization of resources and increased latency. However, the proposed trust-value based check pointing scheme captures a check point only after a node has endured certain movements. Hence, whenever a node move from one cluster to another, the node moves from a cluster with high trust value to a cluster with low trust value and vice-versa. Therefore, nodes do not rely on any fixed threshold value in order to take a checkpoint. This is due to dependency of trust value of a node on the previous and current trust value of its cluster. Thus, each time a node moves or change to a new cluster, its cluster_change_count is compared with a fixed threshold value. The information, hence, is restrained for communication ifthe count is greater than threshold in order to maintain its safety. The proposed algorithm comprises of three phases, namely, multi-checkpointing phase , trust node evaluation phase and recovery phase . The analysis shows better performance of the proposed protocol over existing mobility- based protocol in terms of probability of recovery of failed nodes, residual energy and simulation time.

Unified model for assessing checkpointing protocols at extreme‐scale

SUMMARYIn this paper, we present a unified model for several well‐known checkpoint/restart protocols. The proposed model is generic enough to encompass both extremes of the checkpoint/restart space, from coordinated approaches to a variety of uncoordinated checkpoint strategies (with message logging). We identify a set of crucial parameters, instantiate them, and compare the expected efficiency of the fault tolerant protocols, for a given application/platform pair. We then propose a detailed analysis of several scenarios, including some of the most powerful currently available high performance computing platforms, as well as anticipated Exascale designs. The results of this analytical comparison are corroborated by a comprehensive set of simulations. Altogether, they outline comparative behaviors of checkpoint strategies at very large scale, thereby providing insight that is hardly accessible to direct experimentation. Copyright © 2013 John Wiley & Sons, Ltd.

A Heuristic Approach for the Automatic Insertion of Checkpoints in Message-Passing Codes

Abstract: Checkpointingtoolsmaybetypically implementedattwodifferentabstrac-tion levels: at the system level or at the application level. The latter has become a morepopular alternative due to its flexibility and the possibility of operating in different en-vironments. However, application-level checkpointing tools often require the user tomanually insert checkpoints in order to ensure that certain requirements are met (e.g.forcing checkpoints to be taken at the user code and not inside kernel routines). Theapproach presented in this work is twofold. First, a spatial coordination protocol forcheckpointing parallel SPMD applications is proposed, based on forcing checkpoints tobe taken at the same places in the application code by all processes. Thus, global con-sistency is achieved without adding any new runtime communications or piggybackeddata, and without the need to use specific fault-tolerant message-passing implemen-tations. Second, the paper also introduces a compilation technique for the automaticinsertion of checkpoints using the spatial coordination protocol, based on a static anal-ysis of communications and a heuristic analysis of computational load. These analysescan also be used to achieve automatic checkpoint insertion in approaches based onclassical protocols, such as uncoordinated checkpointing or distributed snapshots.Key Words: Fault tolerance, checkpointing, parallel programming, message-passing,compiler-supportCategory: C.4, D.1.3

MPICH-V Project: A Multiprotocol Automatic Fault-Tolerant MPI

High performance computing platforms such as Clusters, Grid and Desktop Grids are becoming larger and subject to more frequent failures. MPI is one of the most used message passing libraries in HPC applications. These two trends raise the need for fault-tolerant MPI. The MPICH-V project focuses on designing, implementing and comparing several automatic fault-tolerant protocols for MPI applications. We present an extensive related work section highlighting the originality of our approach and the proposed protocols. We then present four fault-tolerant protocols implemented in a new generic framework for fault-tolerant protocol comparison, covering a large spectrum of known approaches from coordinated checkpoint, to uncoordinated checkpoint associated with causal message logging. We measure the performance of these protocols on a micro-benchmark and compare them with the NAS benchmark, using an original fault tolerance test. Finally, we outline the lessons learned from this in depth fault-tolerant protocol comparison of MPI applications.

Coordinated checkpoint versus message log for fault tolerant MPI

Large clusters, high availability clusters and grid deployments often suffer from network, node or operating system faults and thus require the use of fault tolerant programming models. MPI is one of the most widely adopted programming models for high performance computing. There are several approaches for fault tolerance in an MPI environment. The automatic and transparent ones are based on either coordinated or uncoordinated checkpoint associated with a message log strategy. There are many protocols and optimisations for these approaches and several implementations have been made. However, few results of comparison between them exist. Coordinated checkpoint has the advantage of a very low overhead as long as the execution stays fault free. In contrast, uncoordinated checkpoint must be complemented by a message log protocol which adds a significant penalty for all message transfers even for fault free executions. The drawbacks of coordinated checkpoint are the synchronisation cost before the checkpoint, the synchronised checkpoint cost and the restart cost after a fault. Message log does not suffer from these problems, as it processes checkpoint and restart independently. These differences suggest that the best approach depends on the fault frequency. This paper investigates this question from a fair experimental protocol: we implement and test two protocols (coordinated checkpoint and pessimistic message log) on the same system and we compare them on a cluster according to the frequency of faults that are generated artificially. The main conclusion is that uncoordinated checkpoint is relevant for a large scale cluster from one fault every hour for applications with large dataset.

Garbage collection in uncoordinated checkpointing algorithms

In this paper, the hard problem of the thorough garbage collection in uncoordinated checkpointing algorithms is studied. After introduction of the traditional garbage collecting scheme, with which only obsolete checkpoints can be discarded, it is shown that this kind of traditional method may fail to discard any checkpoint in some special cases, and it is necessary and urgent to find a thorough garbage collecting method, with which all the checkpoints useless for any future rollback-recovery including the obsolete ones can be discarded. Then, the Thorough Garbage Collection Theorem is proposed and proved, which ensures the feasibility of the thorough garbage collection, and gives the method to calculate the set of the useful checkpoints as well.

Checkpoint space reclamation for uncoordinated checkpointing in message-passing systems

Uncoordinated checkpointing allows process autonomy and general nondeterministic execution, but suffers from potential domino effects and the associated space overhead. Previous to this research, checkpoint space reclamation had been based on the notion of obsolete checkpoints; as a result, a potentially unbounded number of nonobsolete checkpoints may have to be retained on stable storage. In this paper, we derive a necessary and sufficient condition for identifying all garbage checkpoints. By using the approach of recovery line transformation and decomposition, we develop an optimal checkpoint space reclamation algorithm and show that the space overhead for uncoordinated checkpointing is in fact bounded by N(N+1)/2 checkpoints where N is the number of processes.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Consistent global checkpoints based on direct dependency tracking

We model the consistent global checkpoints in a distributed system as the maximum-sized antichains in the partially ordered set generated by the happened before relation. While the partially ordered set generated by the direct dependency tracking in uncoordinated checkpointing protocols does not capture the complete happened before relation among all checkpoints, we show that the missing happened before information does not affect the identification of consistent global checkpoints.

Manetho: transparent roll back-recovery with low overhead, limited rollback, and fast output commit

Manetho is a new transparent rollback-recovery protocol for long-running distributed computations. It uses a novel combination of antecedence graph maintenance, uncoordinated checkpointing, and sender-based message logging. Manetho simultaneously achieves the advantages of pessimistic message logging, namely limited rollback and, fast output commit, and the advantage of optimistic message logging, namely low failure-free overhead. These advantages come at the expense of a complex recovery scheme. >