Checkpoint Interval Research Articles

Performance optimization of parallel-distributed processing with checkpointing for cloud environment

In cloud computing, the most successful application framework is parallel-distributed processing, in which an enormous task is split into a number of subtasks and those are processed independently on a cluster of machines referred to as workers. Due to its huge system scale, worker failures occur frequently in cloud environment and failed subtasks cause a large processing delay of the task. One of schemes to alleviate the impact of failures is checkpointing method, with which the progress of a subtask is recorded as checkpoint and the failed subtask is resumed by other worker from the latest checkpoint. This method can reduce the processing delay of the task. However, frequent checkpointing is system overhead and hence the checkpoint interval must be set properly. In this paper, we consider the optimal number of checkpoints which minimizes the task-processing time. We construct a stochastic model of parallel-distributed processing with checkpointing and approximately derive explicit expressions for the mean task-processing time and the optimal number of checkpoints. Numerical experiments reveal that the proposed approximations are sufficiently accurate on typical environment of cloud computing. Furthermore, the derived optimal number of checkpoints outperforms the result of previous study for minimizing the task-processing time on parallel-distributed processing.

The value of real-time continuous glucose monitoring in premature infants of diabetic mothers.

To determine the feasibility of using a real-time continuous glucose monitoring system (RTGMS) in intensive care units, our study focus on preterm infants with diabetic mothers owing to their high risk of blood sugar abnormalities. Thirty preterm babies (M = 15 and F = 15; ≤ 36 week gestation age) were studied from within 72 hours of delivery. These babies were admitted to the newborn intensive care and were further categorized into groups based on whether their mothers with or without diabetic mellitus. Blood sugar levels were monitored by both RTGMS and the traditional intermittent arterial line (A-Line) glucose method. Continuous glucose monitoring were well tolerated in 30 infants. There were good consistency between RTGMS and A-Line glucose concentration measurements. Of the preterm infants, 33.33% experienced abnormal glucose levels (hypoglycemia or hyperglycemia) between the checkpoint intervals of the intermittent A-Line blood sugar measurements. RTGM showed advantages with regards to reduced pain, greater comfort, the provision of real-time information, high sensitivity (94.59%) and specificity (97.87%) in discovering abnormalities of blood sugar, which are especially valuable for premature infants of diabetic mothers. RTGMS is comparable to A-line measurement for identifying fluctuations in blood glucose in premature infants. RTGMS detects more episodes of abnormal glucose concentration than intermittent A-line blood glucose measurement. High risk infants, especially premature infants with diabetic mothers, should receive more intensive blood sugar level checks by using continuous RTGMS.

The impact of checkpointing interval selection on the scheduling performance of real‐time fine‐grained parallel applications in SaaS clouds under various failure probabilities

SummaryAs the adoption of Software as a Service (SaaS) cloud computing continues to gain momentum, the arising challenges of scheduling parallel applications on such platforms need to be addressed. Due to the complexity and the fine‐grained parallelism of the workload, as well as the multi‐tenancy of the underlying host environment, end‐user applications are usually prone to transient software failures. Therefore, fault tolerance is one of the most crucial aspects of scheduling in SaaS clouds. It is usually achieved through application‐directed checkpointing. However, selecting an appropriate checkpointing interval is not a trivial task. Unnecessary frequent checkpointing may degrade the system performance. On the other hand, infrequent checkpointing may lead to greater recovery time and thus poorer performance. Consequently, the checkpointing interval must be selected taking into account the failure probability, as well as the nature of the workload. Towards this direction, we investigate via simulation the impact of checkpointing interval selection on the performance of a SaaS cloud, where fine‐grained parallel applications with firm deadlines and approximate computations are scheduled for execution, under various failure probabilities. The simulation results are analyzed, in an attempt to shed light on the relation between the checkpointing interval and failure probability.

The Optimal Checkpoint Interval for the Long-Running Application

For the distributed computing system, excessive or deficient checkpointing operations would result in severe performance degradation. To minimize the expected computation execution of the long-running application with a general failure distribution, the optimal equidistant checkpoint interval for fault tolerant performance optimization is analyzed and derived in this paper. More precisely, the optimal checkpointing period to determine the proper checkpoint sequence is proposed, and the derivation of the expected effective rate of the defined computation cycle is introduced. Corresponding to the maximal expected effective rate, the constraint of the optimal checkpoint sequence can be obtained. From the constraint of optimality, the optimal equidistant checkpoint interval can be obtained according to the minimal fault tolerant overhead ratio. By the numerical results, the proposal is practical to determine a proper equidistant checkpoint interval for fault tolerant performance optimization.

Unified fault-tolerance framework for hybrid task-parallel message-passing applications

We present a unified fault-tolerance framework for task-parallel message-passing applications to mitigate transient errors. First, we propose a fault-tolerant message-logging protocol that only requires the restart of the task that experienced the error and transparently handles any message passing interface calls inside the task. In our experiments we demonstrate that our fault-tolerant solution has a reasonable overhead, with a maximum observed overhead of 4.5%. We also show that fine-grained parallelization is important for hiding the overheads related to the protocol as well as the recovery of tasks. Secondly, we develop a mathematical model to unify task-level checkpointing and our protocol with system-wide checkpointing in order to provide complete failure coverage. We provide closed formulas for the optimal checkpointing interval and the performance score of the unified scheme. Experimental results show that the performance improvement can be as high as 98% with the unified scheme.

Analytic Model for Optimal Checkpoints in Mobile Real-time Systems

It is not practically feasible to apply hardware-based fault-tolerant schemes, such as hardware replication, in mobile devices. Therefore, software-based fault-tolerance techniques, such as checkpoint and rollback schemes, are required. In checkpoint and rollback schemes, the optimal checkpoint interval should be applied to obtain the best performance. Most previous studies focused on minimizing the expected execution time or response time for completing a given task. Currently, most mobile applications run in real-time environments. Therefore, it is extremely essential for mobile devices to employ optimal checkpoint intervals as determined by the real-time constraints of tasks. In this study, we tackle the problem of determining the optimal inter-checkpoint interval of checkpoint and rollback schemes to maximize the deadline meet ratio in real-time systems and to build a probabilistic cost model. From this cost model, we can numerically find the optimal checkpoint interval using mathematical tools. The performance of the proposed solution is evaluated using analytical estimates.

Two-State Checkpointing for Energy-Efficient Fault Tolerance in Hard Real-Time Systems

Checkpointing with rollback recovery is a well-established technique to tolerate transient faults. However, it incurs significant time and energy overheads, which go wasted in fault-free execution states and may not even be feasible in hard real-time systems. This paper presents a low-overhead two-state checkpointing (TsCp) scheme for fault-tolerant hard real-time systems. It differentiates between the fault-free and faulty execution states and leverages two types of checkpoint intervals for these two different states. The first type is nonuniform intervals that are used while no fault has occurred. These intervals are determined based on postponing checkpoint insertions in fault-free states, with the aim of decreasing the number of checkpoint insertions. The second type is uniform intervals that are used from the time when the first fault occurs. They are determined so as to minimize execution time for faulty states, leaving more time available for energy management in fault-free states. Experimental evaluation on an embedded processor (LEON3) and an emerging nonvolatile memory technology (ReRAM) illustrates that TsCp significantly reduces the number of checkpoints (62% on average) compared with previous works, while preserving fault tolerance. This results in 14% and 13% reduced execution time and energy consumption, respectively. Furthermore, we combine TsCp with dynamic voltage scaling (DVS) and achieve up to 26% (21% on average) energy saving compared with the state-of-the-art techniques.

분산 고장 탐지 방식을 이용한 실시간 태스크에서의 최적 체크포인터 구간 선정

체크포인터를 삽입한 실시간 시스템에서는 고장이 발생하면 고장 직전의 체크포인터로 회귀하여 태스크를 재실행함으로써 과도 고장을 효과적으로 극복할 수 있다. 이번 논문에서는 체크포인터에서 실행되는 데이터 저장과 고장 탐지 과정을 분리한 새로운 체크포인터 방식을 제안한다. 하나의 체크포인터 구간 내에 여러 개의 고장 탐지 과정을 추가하면 고장 발생에서 탐지까지의 지연 시간을 줄일 수 있다. 본 논문에서는 태스크가 데드라인 이내에서 성공적으로 수행될 확률을 최대화하는 고장 탐지 과정의 삽입 방법을 제안한다. 고장 탐지 과정이 분리된 체크포인터 방식을 마코프 체인으로 모델링하고 실시간 태스크의 성공적 수행 확률을 계산하는 모의실험을 수행하여 최적의 해를 구하는 과정을 제시한다. Checkpoint placement is an effective fault tolerance technique against transient faults in which the task is re-executed from the latest checkpoint when a fault is detected. In this paper, we propose a new checkpoint placement strategy separating data saving and fault detection processes that are performed together in conventional checkpoints. Several fault detection processes are performed in one checkpoint interval in order to decrease the latency between the occurrence and detection of faults. We address the placement method of fault detection processes to maximize the probability of successful execution of a task within the given deadline. We develop the Markov chain model for a real-time task having the proposed checkpoints, and derive the optimal fault detection and checkpoint interval.

Provenance Based Checkpointing Method for Dynamic Health Care Smart System

Smart systems in telemedicine frequently use intelligent sensor devices at large scale. Practitioners can monitor non-stop the vital parameters of hundreds of patients in real-time. The most important pillars of remote patient monitoring services are communication and data processing. Large scale data processing is done mainly using work flows. Some work flows are working in real-time, more complex ones are running for days or even for weeks on parallel and distributed infrastructures such as HPC systems and cloud. In HPC environment high number of failures can arise during health care smart systems work ow enactment, so the use of fault tolerance techniques is unavoidable. The most frequently used fault tolerance technique is checkpointing. The effectiveness of the checkpointing method depends on the checkpointing interval. In this work we give a brief overview of the different checkpointing techniques and propose two new provenance based checkpointing algorithms which uses the information stored in the work ow structure to dynamically change the frequency of checkpointing and can be efficiently used for dynamic health care smart systems.

Adaptive Remus: adaptive checkpointing for Xen-based virtual machine replication

With the ever increasing dependence on computers and networks, many systems are required to be continuously available in order to fulfil their mission. Virtualization technology enables high availability to be offered in a convenient, cost-effective manner: with the encapsulation provided by virtual machines (VMs), entire systems can be replicated transparently in software, obviating the need for expensive fault-tolerant hardware. Remus is a VM replication mechanism for the Xen hypervisor that provides high availability despite crash failures. Replication is performed by checkpointing the VM at fixed intervals. However, there is an antagonism between processing and communication regarding the optimal checkpoint interval: while longer intervals benefit processor-intensive applications, shorter intervals favour network-intensive applications. Thus, any chosen interval may not always be suitable for the hosted applications, limiting Remus usage in many scenarios. This work introduces Adaptive Remus, a proposal for adaptive checkpointing in Remus that dynamically adjusts the replication frequency according to the characteristics of running applications. Experimental results indicate that our proposal improves performance for applications that require both processing and communication, without harming applications that use only one type of resource.

Toward an Optimal Online Checkpoint Solution under a Two-Level HPC Checkpoint Model

The traditional single-level checkpointing method suffers from significant overhead on large-scale platforms. Hence, multilevel checkpointing protocols have been studied extensively in recent years. The multilevel checkpoint approach allows different levels of checkpoints to be set (each with different checkpoint overheads and recovery abilities), in order to further improve the fault tolerance performance of extreme-scale HPC applications. How to optimize the checkpoint intervals for each level, however, is an extremely difficult problem. In this paper, we construct an easy-to-use two-level checkpoint model. Checkpoint level 1 deals with errors with low checkpoint/recovery overheads such as transient memory errors, while checkpoint level 2 deals with hardware crashes such as node failures. Compared with previous optimization work, our new optimal checkpoint solution offers two improvements: (1) it is an online solution without requiring knowledge of the job length in advance, and (2) it shows that periodic patterns are optimal and determines the best pattern. We evaluate the proposed solution and compare it with the most up-to-date related approaches on an extreme-scale simulation testbed constructed based on a real HPC application execution. Simulation results show that our proposed solution outperforms other optimized solutions and can improve the performance significantly in some cases. Specifically, with the new solution the wall-clock time can be reduced by up to 25.3 percent over that of other state-of-the-art approaches. Finally, a brute-force comparison with all possible patterns shows that our solution is always within 1 percent of the best pattern in the experiments.

A novel adaptive checkpointing method based on information obtained from workflow structure

Scientific workflows are data- and compute-intensive; thus, they may run for days or even weeks on parallel and distributed infrastructures such as grids, supercomputers, and clouds. In these high-performance computing infrastructures, the number of failures that can arise during scientific-workflow enactment can be high, so the use of fault-tolerance techniques is unavoidable. The most-frequently used fault-tolerance technique is taking checkpoints from time to time; when failure is detected, the last consistent state is restored. One of the most-critical factors that has great impact on the effectiveness of the checkpointing method is the checkpointing interval. In this work, we propose a Static (Wsb) and an Adaptive (AWsb) Workflow Structure Based checkpointing algorithm. Our results showed that, compared to the optimal checkpointing strategy, the static algorithm may decrease the checkpointing overhead by as much as 33% without affecting the total processing time of workflow execution. The adaptive algorithm may further decrease this overhead while keeping the overall processing time at its necessary minimum.

Checkpointing with Minimal Recovery in Adhoc Net Based TMR

This paper describes two-fold approach towards utilizing Triple Modular Redundancy (TMR) in Wireless Adhoc Network (AdocNet). A distributed checkpointing and recovery protocol is proposed. The protocol eliminates useless checkpoints and helps in selecting only dependent processes in the concerned checkpointing interval, to recover. A process starts recovery from its last checkpoint only if it finds that it is dependent (directly or indirectly) on the faulty process. The recovery protocol also prevents the occurrence of missing or orphan messages. In AdocNet, a set of three nodes (connected to each other) is considered to form a TMR set, being designated as main, primary and secondary. A main node in one set may serve as primary or secondary in another. Computation is not triplicated, but checkpoint by main is duplicated in its primary so that primary can continue if main fails. Checkpoint by primary is then duplicated in secondary if primary fails too.

Schedulability of Fault Tolerant Real Time System Based on Local Optimum Checkpoint Under Priority Mixed Strategy

Against the defect of the real time system fault tolerant model that can only tolerate one fault, the Local optimum checkpoint (LOC) algorithm was proposed. Then according to the schedulability analysis, the worst case response time formula under the fault tolerant priority mixed strategy based on the local optimum checkpoint was deduced. Finally the Fault tolerant priority configuration search algorithm under Mixed strategy based on LOC (FTPCS-MS-LOC) was proposed. The FTPCS-MS-LOC can toleratemultiple transient faults. The algorithm can effectively reduce the search space compared to the enumeration method. The simulation shows that the FTPCS-MSLOC can significantly improve the system fault resilience than the fault tolerant priority inheritance, promotion and demotion strategy, and also the mixed strategy based on the checkpoint interval in the condition of single fault occurred.

VM-μCheckpoint: Design, Modeling, and Assessment of Lightweight In-Memory VM Checkpointing

Checkpointing and rollback techniques enhance reliability and availability of virtual machines and their hosted IT services. This paper proposes VM-μCheckpoint, a light-weight pure-software mechanism for high-frequency checkpointing and rapid recovery for VMs. Compared with existing techniques of VM checkpointing, VM-μCheckpoint tries to minimize checkpoint overhead and speed up recovery by means of copy-on-write, dirty-page prediction and in-place recovery, as well as saving incremental checkpoints in volatile memory. Moreover, VM-μCheckpoint deals with the issue that latency in error detection potentially results in corrupted checkpoints, particularly when checkpointing frequency is high. We also constructed Markov models to study the availability improvements provided by VM-μCheckpoint (from 99 to 99.98 percent on reasonably reliable hypervisors). We designed and implemented VM-μCheckpoint in the Xen VMM. The evaluation results demonstrate that VM-μCheckpoint incurs an average of 6.3 percent overhead (in terms of program execution time) for 50 ms checkpoint intervals when executing the SPEC CINT 2006 benchmark. Error injection experiments demonstrate that VM-μCheckpoint, combined with error detection techniques in RMK, provides high coverage of recovery.

Adaptive Checkpoint Interval Algorithm Considering Task Deadline and Lifetime Reliability for Real-Time System

Abstract Checkpointing mechanism is used to tolerate the impact of transient faults by rollback operation. Recently, it has also been used as a mechanism to enhance system's lifetime by identifying and tolerating permanent fault 5,19,10,12. However, equidistant checkpoint interval may cause task deadline violation in the system. Here, we propose an adaptive checkpoint interval placement algorithm (ADeLiRACI) that meets all tasks deadline. The checkpoint intervals are adjusted to minimize the impact of stresses and permanent faults on the running hosts. This novel mechanism allows greater applicability in real time systems with hard deadline such as weather prediction, financial transactions etc. We compare the estimated completion time for increasing fault-rate in the system against five existing algorithms. For all applications, ADeLiRACI is able to meet the hard deadline along with enhancing lifetime reliability of the system.

GloudSim: Google trace based cloud simulator with virtual machines

SummaryIn 2011, Google released a 1‐month production trace with hundreds of thousands of jobs running across over 12,000 heterogeneous hosts. In order to perform in‐depth research based on the trace, it is necessary to construct a close‐to‐practice simulation system. In this paper, we devise a distributed cloud simulator (or toolkit) based on virtual machines, with three important features. (1) The dynamic changing resource amounts (such as CPU rate and memory size) consumed by the reproduced jobs can be emulated as closely as possible to the real values in the trace. (2) Various types of events (e.g., kill/evict event) can be emulated precisely based on the trace. (3) Our simulation toolkit is able to emulate more complex and useful cases beyond the original trace to adapt to various research demands. We evaluate the system on a real cluster environment with 16×8=128 cores and 112 virtual machines constructed by XEN hypervisor. To the best of our knowledge, this is the first work to reproduce Google cloud environment with real experimental system setting and real‐world large scale production trace. Experiments show that our simulation system could effectively reproduce the real checkpointing/restart events based on Google trace, by leveraging Berkeley Lab Checkpoint/Restart tool. It can simultaneously process up to 1200 emulated Google jobs over the 112 virtual machines. Such a simulation toolkit has been released as a GNU GPL v3 software for free downloading, and it has been successfully applied to the fundamental research on the optimization of checkpoint intervals for Google tasks. Copyright © 2014 © Published 2014. This article is a U.S. Government work and is in the public domain in the USA.

Hardware Fault Recovery for I/O Intensive Applications

With continued process scaling, the rate of hardware failures in commodity systems is increasing. Because these commodity systems are highly sensitive to cost, traditional solutions that employ heavy redundancy to handle such failures are no longer acceptable owing to their high associated costs. Detecting such faults by identifying anomalous software execution and recovering through checkpoint-and-replay is emerging as a viable low-cost alternative for future commodity systems. An important but commonly ignored aspect of such solutions is ensuring that external outputs to the system are fault-free. The outputs must be delayed until the detectors guarantee this, influencing fault-free performance. The overheads for resiliency must thus be evaluated while taking these delays into consideration; prior work has largely ignored this relationship. This article concerns recovery for I/O intensive applications from in-core faults. We present a strategy to buffer external outputs using dedicated hardware and show that checkpoint intervals previously considered as acceptable incur exorbitant overheads when hardware buffering is considered. We then present two techniques to reduce the checkpoint interval and demonstrate a practical solution that provides high resiliency while incurring low overheads.

CONSISTENCY OF DISTRIBUTED SYSTEM WITH ACTIVE INITIATOR PROCESS WITHOUT USELESS CHECKPOINTS

Checkpointing mechanism is the one of the best attractive approach for providing software fault tolerance in distributed message passing systems. This paper aims to implement a distributed checkpointing technique, which eliminates the drawbacks of the centralized approach like “domino effect”, “useless checkpoint” (checkpoints that do not contribute to global consistency), and “hidden and zigzag” dependencies. The proposed checkpointing protocol has a checkpoint initiator, but, coordination among the local checkpoints is done in a distributed fashion. This guaranty that no message would be lost in case of failure occurs, has been maintained in this work by exchange of information among the processes. However, there is no central checkpoint initiator, but each of the processes takes turn to act as an initiator. Processes take local checkpoints only after being notified by the initiator. The processes synchronize their activities of the current checkpointing interval before finally committing their checkpoints. Thus, the checkpointing pattern described in this paper takes only those checkpoints that will contribute to the consistent global snapshot thereby eliminating the number of useless checkpoints.

Stability Assessment Metamorphic Approach (SAMA) for Effective Scheduling based on Fault Tolerance in Computational Grid

Grid Computing allows coordinated and controlled resource sharing and problem solving in multi-institutional, dynamic virtual organizations. Moreover, fault tolerance and task scheduling is an important issue for large scale computational grid because of its unreliable nature of grid resources. Commonly exploited techniques to realize fault tolerance is periodic Checkpointing that periodically saves the job’s state. But an inappropriate checkpointing interval prevails to delay in the job execution, and reduces the throughput. With that concern, this paper endeavors to ensure better performance on computational grid with more effective and reliable fault tolerant system using a novel Stability Assessment Metamorphic Approach (SAMA). Here, the strategy used to attain fault tolerance is by adapting the checkpoints depending on the current status and past failure information of the resources dynamically, which is being maintained in the information server. Effective scheduling process can be achieved by fault tolerance based scheduling that involves in determination of deviation rate of all nodes using some high-stability assessment constraints. This evinces the job to be accomplished within the deadline with improved throughput and paves a way for making the grid environment trust worthy.