Application-level Fault Tolerance Research Articles

A Comparison of Application-Level Fault Tolerance Schemes for Task Pools

Fault tolerance is an important requirement for successful program execution on exascale systems. The common approach, checkpointing, regularly saves a program’s state, such that the execution can be restarted after permanent node failures. Checkpointing is often performed on system level, but its deployment on application level can reduce the running time overhead.The drawback of application-level checkpointing is a higher programming expense. It pays off if the checkpointing is applied to reusable patterns. We consider task pools, which exist in many variants. The paper supposes that tasks are generated dynamically and are free of side effects. Further, the final result must be computed from individual task results by reduction. Moreover, the pools must be distributed with private queues, and adopt work stealing.The paper describes and evaluates three application-level fault tolerance schemes for task pools. All use uncoordinated checkpointing and regularly save information in a resilient store. The first scheme (called AllFT) saves descriptors of all open tasks; the second scheme (called IncFT) selectively and incrementally saves only part of them; and the third scheme (called LogFT) logs stealing events and writes checkpoints in parallel to task processing.All schemes have been implemented by extending the Global Load Balancing (GLB) library of the “APGAS for Java” programming system. In experiments with the UTS, NQueens, and BC benchmarks with up to 672 workers, the running time overhead during failure-free execution, compared to a non-resilient version of GLB, was typically below 6%. The recovery cost was negligible, and there was no clear winner among the three schemes. A more detailed performance analysis with synthetic benchmarks revealed that IncFT and LogFT are superior in scenarios with large task descriptors.

Analytics of resource transients in cloud-based applications

Guaranteeing QoS of services deployed in clouds is a key issue in cloud environments. Cloud services rely heavily on sharing resources between tenants; users have only partial knowledge and control of them. Limited observability and controllability make guaranteeing QoS a challenging task. Many important classic approaches to QoS assurance cannot be adopted, as for a tenant resource arbitration is a black box. Unexpected changes in resource allowance and characteristics introduce novel risks for demanding applications, most importantly soft real-time ones. We introduce the concept of applicationintegrated early warning sensors – ‘mystery shoppers’ – to increase the observability of the platform performability state. Mystery shoppers are special, continuously running benchmarks that run as low footprint applications. They deliver fine-grained reports by simulating application resource usage and measuring platform characteristics. This enables inferring the hidden resource sharing characteristics of public clouds. The derived metrics can aid deployment planning and control application-level fault tolerance mechanisms.

A survey of linguistic structures for application-level fault tolerance

Structures for the expression of fault-tolerance provisions in application software comprise the central topic of this article. Structuring techniques answer questions as to how to incorporate fault tolerance in the application layer of a computer program and how to manage the fault-tolerant code. As such, they provide the means to control complexity, the latter being a relevant factor for the introduction of design faults. This fact and the ever-increasing complexity of today's distributed software justify the need for simple, coherent, and effective structures for the expression of fault-tolerance in the application software. In this text we first define a “base” of structural attributes with which application-level fault-tolerance structures can be qualitatively assessed and compared with each other and with respect to the aforementioned needs. This result is then used to provide an elaborated survey of the state-of-the-art of application-level fault-tolerance structures.

Performance tuning policies for application level fault tolerance in distributed object systems

In distributed object systems, application level fault tolerance is often attained by appropriate object replication policies. These policies aim at increasing the exhibited service availability by masking potential faults that do not recur after recovery. Existing middleware support infrastructures allow customizing object replication properties. However, since fault tolerance has a significant impact in the perceived service performance, there is a need for a suitable quantitative design technique, which allows comparing different replication policies by trading off the caused overhead cost against the achieved fault-tolerance effectiveness. We are also interested in taking into account different concerns in a combined manner (e.g. fault tolerance combined with load balancing and multithreading). This paper presents experimental evidence for the most important performance tradeoffs revealed in a simulation-based study. We considered different cases of object request loss behavior for the faulty objects, as well as, a number of request-retry strategies. The experiments took place in two different application workload levels for varied fault detection settings. We provide results for the combined effects of the studied replication policies with two specific load-balancing strategies. The presented results constitute a valuable experience report for performance tuning object replication policies for application level fault tolerance.

Process Fault Tolerance: Semantics, Design and Applications for High Performance Computing

With increasing numbers of processors on current machines, the probability for node or link failures is also increasing. Therefore, application-level fault tolerance is becoming more of an important issue for both end-users and the institutions running the machines. In this paper we present the semantics of a fault-tolerant version of the message passing interface (MPI), the de-facto standard for communication in scientific applications, which gives applications the possibility to recover from a node or link error and continue execution in a well-defined way. We present the architecture of fault-tolerant MPI, an implementation of MPI using the semantics presented above as well as benchmark results with various applications. An example of a fault-tolerant parallel equation solver, performance results as well as the time for recovering from a process failure are furthermore detailed.

Application-Level Fault Tolerance as a Complement to System-Level Fault Tolerance

As multiprocessor systems become more complex, their reliability will need to increase as well. In this paper we propose a novel technique which is applicable to a wide variety of distributed real-time systems, especially those exhibiting data parallelism. System-level fault tolerance involves reliability techniques incorporated within the system hardware and software whereas application-level fault tolerance involves reliability techniques incorporated within the application software. We assert that, for high reliability, a combination of system-level fault tolerance and application-level fault tolerance works best. In many systems, application-level fault tolerance can be used to bridge the gap when system-level fault tolerance alone does not provide the required reliability. We exemplify this with the RTHT target tracking benchmark and the ABF beamforming benchmark.

Application Level Fault Tolerance in Heterogeneous Networks of Workstations

We have explored methods for checkpointing and restarting processes within the distributed object migration environment (Dome), a C++ library of data parallel objects that are automatically distributed over heterogeneous networks of workstations (NOWs). System level checkpointing methods, although transparent to the user, were rejected because they lack support for heterogeneity. We have implemented application level checkpointing which places the checkpoint and restart mechanisms within Dome's C++ objects. Application level checkpointing has been implemented with a library-based technique for the programmer and a more transparent preprocessor-based technique. Dome's implementation of checkpointing successfully checkpoints and restarts processes on different numbers of machines and different architectures. Results from executing Dome programs across a NOW with realistic failure rates have been experimentally determined and are compared with theoretical results. The overhead of checkpointing is found to be low, while providing substantial decreases in expected runtime on realistic systems.

Snooping fault-tolerant distributed shared memories

Distributed-memory systems that have no physically shared memory can be programmed by use of a shared-memory model by simulating a virtual shared address space. These distributed shared memory (DSM) systems can be easier to program than systems that use a message-passing model. However, as the number of processors implementing the DSM grows, the probability of failure grows. This article presents a set of fault-tolerant DSM (FTDSM) algorithms. The algorithms are derived from a system consisting of a set of processors connected by a broadcast network with one special processor, the snooper, that monitors network traffic. The snooper keeps a backup of the current state of the shared memory and can respond for failed processors. The addition of the snooper improves the reliability of the DSM at a lower cost than making each of the other processors fault tolerant. It is also shown how different degrees of application level fault tolerance can be incorporated into the FTDSM protocol to allow the recovery of application processes. The snooping FTDSM is extended to include multiple snoopers; then an integrated snooper FTDSM is developed that is not restricted to a broadcast network.