Checkpoint Interval Research Articles

Checkpointing models for tasks of different types

A server subject to random breakdowns and repairs offers services to incoming jobs whose lengths are highly variable. A checkpointing policy is in operation, aiming to protect against possibly lengthy recovery periods by backing up the current state at periodic checkpoints. The problem of how to choose a checkpointing interval to optimise performance is addressed by analysing a general queueing model which includes breakdowns, repairs, back-ups and recoveries. Exact solutions are obtained under both Markovian and non-Markovian assumptions. Numerical experiments illustrate the conditions where checkpoints are useful and where they are not, and, in the former case, quantify the achievable benefits.

A Dynamic Checkpoint Interval Decision Algorithm for Live Migration-Based Drone-Recovery System

Numerous services and applications have been developed to monitor anomalies or collect various sensing information in large-scale monitoring areas using drones. Nonetheless, interruptions of drone missions in such areas occasionally occur due to network errors, low battery levels, or physical defects, such as damage to the rotor and propeller. Checkpointing is a technique that periodically saves the system’s state, allowing it to be restored to that point in the event of a failure. In such circumstances, checkpointing techniques can be used to periodically save information related to the drone mission and replace a malfunctioning drone with the saved checkpoint information. In this paper, we propose a dynamic checkpoint interval decision algorithm for a live migration-based drone-recovery system. The proposed scheme minimizes the drone’s energy consumption while efficiently performing checkpointing. According to the basic experimental results, the proposed scheme consumed only about 3.51% more energy, while performing about 25.97% more checkpoint operations compared to the FIC (Fixed Interval Checkpointing) scheme. By using the proposed scheme, it is possible to increase the availability of checkpoint information and quickly resume drone missions, while minimizing the increase in energy consumption of the drone by saving checkpoints more frequently. Therefore, the proposed scheme can improve the reliability and stability of drone-based services.

In-Memory Versioning (IMV)

In this paper, we propose and evaluate designs for a novel hardware-assisted data versioning system (in-memory versioning or IMV) in the context of high-performance computing. Our main novelty and advantage over recent published work is that it does not require any changes to host processor logic, instead augmenting a memory controller within memory modules. It is faster and more efficient than existing high-performance computing (HPC) checkpointing schemes and works from hours to sub-second checkpoint intervals. The main premise is to perform most operations in hardware at cache-line granularity, avoiding operating system (OS) latency and page copying bandwidth overhead. Energy is saved by keeping data movement in the memory module, compared with page granularity cross channel or cross-network copying that is currently used. For a 1-second checkpoint commit interval, we demonstrate up to 20x checkpoint performance and 70x energy savings using IMV versus page copy-on-write (COW).

Intelligent Fault-Tolerant Mechanism for Data Centers of Cloud Infrastructure

Fault tolerance in cloud computing is considered as one of the most vital issues to deliver reliable services. Checkpoint/restart is one of the methods used to enhance the reliability of the cloud services. However, many existing methods do not focus on virtual machine (VM) failure that occurs due to the higher response time of a node, byzantine fault, and performance fault, and existing methods also ignore the optimization during the recovery phase. This paper proposes a checkpoint/restart mechanism to enhance reliability of cloud services. Our work is threefold: (1) we design an algorithm to identify virtual machine failure due to several faults; (2) an algorithm to optimize the checkpoint interval time is designed; (3) lastly, the asynchronous checkpoint/restart with log-based recovery mechanism is used to restart the failed tasks. The valuation results obtained using a real-time dataset shows that the proposed model reduces power consumption and improves the performance with a better fault tolerance solution compared to the nonoptimization method.

Optimizing checkpoint‐based fault‐tolerance in distributed stream processing systems: Theory to practice

AbstractFault‐tolerance is an essential part of a stream processing system that guarantees data analysis could continue even after failures. State‐of‐the‐art distributed stream processing systems use checkpointing to support fault‐tolerance for stateful computations where the state of the computations is periodically persisted. However, the frequency of performing checkpoints impacts the performance (utilization, latency, and throughput) of the system as the checkpointing process consumes resources and time that can be used for actual computations. In practice, systems are often configured to perform checkpoints based on crude values ignoring factors such as checkpoint and restart costs, leading to suboptimal performance. In our previous work, we proposed a theoretical optimal checkpoint interval that maximizes the system utilization for stream processing systems to minimize the impact of checkpointing on system performance. In this article, we investigate the practical benefits of our proposed theoretical optimal by conducting experiments in a real‐world cloud setting using different streaming applications; we use Apache Flink, a well‐known stream processing system for our experiments. The experiment results demonstrate that an optimal interval can achieve better utilization, confirming the practicality of the theoretical model when applied to real‐world applications. We observed utilization improvements from 10% to 200% for a range of failure rates from 0.3 failures per hour to 0.075 failures per minute. Moreover, we explore how performance measures: latency and throughput are affected by the optimal interval. Our observations demonstrate that significant improvements can be achieved using the optimal interval for both latency and throughput.

Managing Big Interval Data with CINTIA: The Checkpoint INTerval Array

Intervals have become prominent in data management as they are the main data structure to represent a number of key data types such as temporal or genomic data. Yet, there exists no solution to compactly store and efficiently query big interval data. In this paper we introduce CINTIA-the Checkpoint INTerval Index Array-an efficient data structure to store and query interval data, which achieves high memory locality and outperforms state-of-the art solutions. We also propose a low-latency, Big Data system that implements CINTIA on top of a popular distributed file system and efficiently manages large interval data on clusters of commodity machines. Our system can easily be scaled-out and was designed to accommodate large delays between the various components of a distributed infrastructure. We experimentally evaluate the performance of our approach on several datasets and show that it outperforms current solutions by several orders of magnitude in distributed settings.

Dynamic Fault Tolerance Aware Scheduling for Healthcare System on Fog Computing

Internet of Things (IoT) contributes to improve the quality of life as it supports many applications, especially healthcare systems. Data generated from IoT devices is sent to the Cloud Computing (CC) for processing and storage, despite the latency caused by the distance. Because of the revolution in IoT devices, data sent to CC has been increasing. As a result, another problem added to the latency was increasing congestion on the cloud network. Fog Computing (FC) was used to solve these problems because of its proximity to IoT devices, while filtering data is sent to the CC. FC is a middle layer located between IoT devices and the CC layer. Due to the massive data generated by IoT devices on FC, Dynamic Weighted Round Robin (DWRR) algorithm was used, which represents a load balancing (LB) algorithm that is applied to schedule and distributes data among fog servers by reading CPU and memory values of these servers in order to improve system performance. The results proved that DWRR algorithm provides high throughput which reaches 3290 req/sec at 919 users. A lot of research is concerned with distribution of workload by using LB techniques without paying much attention to Fault Tolerance (FT), which implies that the system continues to operate even when fault occurs. Therefore, we proposed a replication FT technique called primary-backup replication based on dynamic checkpoint interval on FC. Checkpoint was used to replicate new data from a primary server to a backup server dynamically by monitoring CPU values of primary fog server, so that checkpoint occurs only when the CPU value is larger than 0.2 to reduce overhead. The results showed that the execution time of data filtering process on the FC with a dynamic checkpoint is less than the time spent in the case of the static checkpoint that is independent on the CPU status.

Minimizing Energy and Computation in Long-Running Software

Long-running software may operate on hardware platforms with limited energy resources such as batteries or photovoltaic, or on high-performance platforms that consume a large amount of energy. Since such systems may be subject to hardware failures, checkpointing is often used to assure the reliability of the application. Since checkpointing introduces additional computation time and energy consumption, we study how checkpoint intervals need to be selected so as to minimize a cost function that includes the execution time and the energy. Expressions for both the program’s energy consumption and execution time are derived as a function of the failure probability per instruction. A first principle based analysis yields the checkpoint interval that minimizes a linear combination of the average energy consumption and execution time of the program, in terms of the classical “Lambert function”. The sensitivity of the checkpoint to the importance attributed to energy consumption is also derived. The results are illustrated with numerical examples regarding programs of various lengths and showing the relation between the checkpoint interval that minimizes energy consumption and execution time, and the one that minimizes a weighted sum of the two. In addition, our results are applied to a popular software benchmark, and posted on a publicly accessible web site, together with the optimization software that we have developed.

Research on Optimal Checkpointing-Interval for Flink Stream Processing Applications

Nowadays various distributed stream processing systems (DSPSs) are employed to process the ever-expanding real-time data. The DSPSs are highly susceptible to system failure, and the fault-tolerance issue is a major problem, which is getting lot of attention nowadays. Flink is a popular streaming computing framework that implements a lightweight, asynchronous checkpoint technique based on the barrier mechanism to ensure high efficiency in analysing the data. In a checkpoint-based fault-tolerance mechanism, a shorter checkpoint interval can increase runtime cost of streaming applications, while a longer one will increase recovery time of failure recovery. So, selecting an optimal checkpoint interval is critical to attain high efficiency of the streaming applications. Traditional optimal checkpoint interval mechanisms usually assume that the checkpointing delay and the fault recovery time are fixed. However, both factors have a strong relation to the intensity of the application’s workload. To obtain more optimal checkpoint interval under different workload intensities, this paper proposes a performance model to estimate the tuples processing latency and a recovery model to estimate the fault recovery time. With these two models, an optimal checkpoint interval can be arrived. These models and the interval optimisation interval are verified experimentally on Flink. The results show that the proposed model can recommend an optimal checkpoint interval according to the system reliability related indicators. This proposed system optimised recovery time and performs efficiently in applications with delay constraints.

Efficient Analysis of Repairable Computing Systems Subject to Scheduled Checkpointing

To improve the success probability of a mission execution, scheduled checkpointing is often implemented to save completed portions of the mission task so that a system can resume the mission execution effectively after its restoration whenever the system failure occurs. This paper considers a repairable computing system subject to the scheduled checkpointing. The checkpointing intervals are deterministic, but can be even or uneven. The system repair time is fixed while the system time-to-failure can follow any arbitrary type of distributions. The maximum number of repairs is specified by a certain threshold value. A multi-valued decision diagram (MDD)-based analytical approach is proposed to evaluate the exact success probability of a mission execution for the considered repairable system. The proposed approach enables generating a compact mission MDD model where identical subMDD models can be merged to improve computational efficiency and reduce storage requirement. The MDD model, once being constructed, can be reused for system reliability evaluations using different input parameter values. A benchmark study is presented to show the efficiency of proposed MDD approach. A case study is performed to illustrate the application of the proposed MDD approach to facilitate decision making about proper system design and parameter selection.

On Providing OS Support to Allow Transparent Use of Traditional Programming Models for Persistent Memory

The advent of persistent memory (PM) into our everyday computing environment is now imminent. New programming models and algorithms based on these models are being developed for such systems. However, current models require programs to be rewritten with persistence related primitives such as clflush and clwb or at least recompiled so that persistent mechanisms can be automatically inserted. This is a burden to program developers. Furthermore, executing legacy programs as-is can lead to application and system inconsistencies as unexpected faults occur. In this article, we propose µSnap, an operating system support that is based on checkpointing that allows legacy applications to be executed as-is without compromising consistency in systems that deploy PM. We implement a prototype of µSnap in the Linux kernel version 4.3.3, and measure and quantify the effect of µSnap for a wide range of applications. We find that µSnap incurs overhead for application execution compared to applications run without any notion of persistency, but that the overhead can be controlled to be minimal by appropriately setting the checkpointing interval. We argue that the benefit for paying this small cost can be tremendous in the sense that one can transparently guarantee the consistency of all legacy software written under the traditional programming model.

A utilization model for optimization of checkpoint intervals in distributed stream processing systems

State-of-the-art distributed stream processing systems such as Apache Flink and Storm have recently included checkpointing to provide fault-tolerance for stateful applications. This is a necessary eventuality as these systems head into the Exascale regime, and is evidently more efficient than replication as state size grows. However current systems use a nominal value for the checkpoint interval, indicative of assuming roughly 1 failure every 19 days, that does not take into account the salient aspects of the checkpoint process, nor the system scale, which can readily lead to inefficient system operation. To address this shortcoming, we provide a rigorous derivation of utilization – the fraction of total time available for the system to do useful work – that incorporates checkpoint interval, failure rate, checkpoint cost, failure detection and restart cost, depth of the system topology and message delay. Our model yields an elegant expression for utilization and provides an optimal checkpoint interval given these parameters, interestingly showing it to be dependent only on checkpoint cost and failure rate. We confirm the accuracy and efficacy of the model through simulations and experiments with Apache Flink. Observations of the simulations validate our theoretical model and demonstrate that utilization can be improved using the derived optimal checkpoint interval. Moreover, experimental results with Apache Flink show that we can obtain improvements in system utilization for every case we tested, especially as the system size increases. Our model provides a solid theoretical basis for the analysis and optimization of more elaborate checkpointing approaches.

Semi-Stream Similarity Join Processing in a Distributed Environment

Similarity join has become very important for semi- or un-structured data processing and analysis. Although several studies have been conducted on the similarity join, little attention has been paid to a semi-stream similarity join, which is a similarity join between stream data and a large disk-based relation. In this study, we propose the first distributed solution called DSim-Join for semi-stream similarity join problem. DSim-Join minimizes the data transmission, reduces database accesses using a cache in a distributed stream processing engine, parallelizes join processing, and balances the load between parallel join threads. Experimental results obtained using real-world datasets show that DSim-Join yields significantly improved throughput compared with state-of-the-art methods, especially for large datasets. The results also show that DSim-Join is scalable and stable; it is not very sensitive to the parameters such as the micro-batch interval, checkpoint interval, and similarity threshold.

An optimal checkpointing model with online OCI adjustment for stream processing applications

SummaryCheckpoint‐based fault‐tolerant (FT) methods have been widely used to enhance the reliability of stream processing systems, but a checkpointing process usually introduces considerable overhead. It is a critical issue to choose the optimal checkpoint interval (OCI) that maximizes the processing efficiency. Traditional OCI models consider the recovery time equals to the execution time from the last checkpoint to the failure moment. However, for stream processing jobs, the recovery time is related to reprocessing workloads, depending on the real‐time input data before a failure. A new model is needed to choose the OCI for stream processing applications. Moreover, the input data rate of a stream processing job fluctuates over time. To solve these problems, we present a novel DSPS OCI (DOCI) model in this paper. We prove that it maximizes the processing efficiency for a given time. We propose an approach to dynamically adjust the OCI for an application to accommodate the workload fluctuations. We conduct simulation experiments to verify the effectiveness of our DOCI model and the efficiency of the online OCI adjustment algorithm. Experimental results with a real‐world dataset show that DOCI achieves an improvement on system efficiency by up to 32%, compared with existing FT approaches.

Instantaneous Mean-Time-To-Failure (MTTF) estimation for checkpoint interval computation at run time

The Mean-Time-To-Failure (MTTF) is an important parameter that determines the life-time reliability of a system. It is being used in several fault-tolerant mechanisms to take a critical decision on processor/system state. Recently it has been found that the MTTF of a system varies with the environmental conditions, in contrary to the earlier belief of a constant MTTF for electronic chips. Thus there is a need for a good and fast estimate of the MTTF that can accommodate the variation of environmental conditions and the stresses on the system. This paper presents an instantaneous MTTF estimation technique to be executed at runtime of the system. A major contribution of this paper is proposing a simple technique to obtain the MTTF for checkpoint interval computation in real-time systems. Our complete system model consisting of multi-level steps are presented as the main model for the MTTF estimation. We adopt one of the state-of-the-art solutions to obtain the aging rate parameter for the host/processor. Also, we proposed another parameter in the MTTF computation that represents the workload and the stress factor of the running host. The results show that the differences are marginal and they lie between 0.014% and 0.131% compared to other MTTF estimation techniques. Also, we showed that the proposed technique is able to capture the temperature variation effect (towards the MTTF value) during several simulated runtime scenarios. The proposed MTTF estimation technique has been incorporated in the life-time reliability-aware checkpointing mechanism and it has been shown to work excellently without violating the task deadlines in all cases.

Checkpointing Strategies for Shared High-Performance Computing Platforms

Input/output (I/O) from various sources often contend for scarcely available bandwidth. For example, checkpoint/restart (CR) protocols can help to ensure application progress in failure-prone environments. However, CR I/O alongside an application's normal, requisite I/O can increase I/O contention and might negatively impact performance. In this work, we consider different aspects (system-level scheduling policies and hardware) that optimize the overall performance of concurrently executing CR-based applications that share I/O resources. We provide a theoretical model and derive a set of necessary constraints to minimize the global waste on a given platform. Our results demonstrate that Young/Daly's optimal checkpoint interval, despite providing a sensible metric for a single, undisturbed application, is not sufficient to optimally address resource contention at scale. We show that by combining optimal checkpointing periods with contention-aware system-level I/O scheduling strategies, we can significantly improve overall application performance and maximize the platform throughput. Finally, we evaluate how specialized hardware, namely burst buffers, may help to mitigate the I/O contention problem. Overall, these results provide critical analysis and direct guidance on how to design efficient, CR ready, large -scale platforms without a large investment in the I/O subsystem.

On the design of reactive approach with flexible checkpoint interval to tolerate faults in cloud computing systems

The likelihood of failures rises in cloud computing systems as a result of their unstable nature. Additionally, the size of a cloud computing system varies with time and thus failures become a common incident. Failures have a high impact on cloud performance and the expected benefits for both customers and providers. Fault tolerance is an essential challenge facing cloud providers in order to mitigate the effects of failures and maintaining the Service Level Agreement (SLA) satisfied. Checkpointing is one of the most known reactive fault tolerance techniques used in distributed computing. However, it can incur considerable overheads that depend on the interval of the checkpoint applied and these overheads put down the performance of the cloud. In this paper, a reactive fault tolerance approach in the context of checkpointing is proposed and evaluated with the aim of getting better performance. The approach depends on applying a flexible interval of the checkpoint to reduce overheads. Simulation experiments indicate superior performance of the approach in terms of power consumption, response time, monetary cost and cloud capacity.

The unexpected virtue of almost: Exploiting MPI collective operations to approximately coordinate checkpoints

SummaryCoordinated checkpoint/restart is currently the dominant approach to mitigating the impact of failures on important scientific applications running on large‐scale distributed systems. However, there is widespread evidence that coordinated checkpointing may no longer be viable on next‐generation systems. Uncoordinated checkpoint/restart attempts to address the shortcomings of coordinated checkpoint/restart by allowing application processes to checkpoint their state independently. However, eliminating coordination may significantly degrade application performance. In this paper, we propose an approach that leverages existing coordination in important scientific applications to approximately coordinate checkpoints. Specifically, we propose to extend MPI implementations to force checkpoints to occur immediately after the completion of a collective operation. We evaluate the performance implications of this approach using an existing validated simulation framework. Our results demonstrate that approximately coordinated checkpointing can significantly improve application performance relative to totally uncoordinated checkpointing. We also show that forcing checkpoints to occur following a collective operation has a small impact on the nominal checkpoint interval for several important workloads. As a whole, the results presented in this paper demonstrate that approximately coordinated checkpointing may provide significant performance benefits without significantly increasing the cost of failure recovery.

Cloud computing and big data: Technologies and applications

Cloud computing and big data: Technologies and applications

Efficient modeling and optimizing of checkpointing in concurrent component-based software systems

A common mechanism to improve availability and performance is checkpointing and rollback. When it is time to checkpoint, a system stores a job’s state to nonvolatile memory, and, when a failure occurs, it rolls back to the latest stored state instead of restarting the job from the beginning, thus improving performance in the presence of failures. Too frequent checkpointing reduces the amount of work to be redone in case of failures but generates excessive overhead, degrading performance. This paper presents a novel and very efficient queuing network model that addresses software component contention for hardware resources and shows how it can be used to model checkpointing in heterogeneous component-based software systems. We validated this model against a previous model, developed by the authors, that used Markov Chains. Our new model is orders of magnitude faster than the previous one and can be used to plan for checkpointing at run-time. As an additional contribution of this paper, we present an optimizer to find, for each software component, the optimal checkpointing interval that minimizes execution time, maximizes availability, or minimizes checkpointing overhead.