Software Fault Tolerance Techniques Research Articles

Achieving 100% availability

Fault tolerance is a key requirement for En Route Automation Modernization (ERAM), the FAA's system that manages En Route air traffic over the USA. A system failure could lead to hundreds of flights being delayed or cancelled. Using experience from earlier systems a set of techniques were built into ERAM at inception, including a hot standby copy of each executable and the latest state checkpointed in disk files. As the system matured through formal testing and operational experience at the first sites (2010 - 2015), the goal of 100% availability was not achieved so additional techniques were added. These included exception safety, runaway process protection, and proactive monitoring of the system to detect defects and often resolve them without the air traffic controllers being aware. With the implementation of these additional techniques the FAA has measured ERAM as 100% available from October 2016 at all 20 operational sites. Software fault tolerance techniques have been well documented [2]; this extended abstract describes the specific techniques that have led to ERAM achieving continuous 24x7 availability for 6 years.

An Adaptive Software Fault-Tolerant Framework for Ubiquitous Vehicular Technologies

The probability of the occurrence of faults increases manifolds when program lines of code exceed a few thousand in ubiquitous applications. Fault mitigation in ubiquitous applications, such as those of autonomous vehicular technologies (VTs), has not been effective even with the use of formal methods. Faults in such applications require exhaustive testing for a timely fix, which seems infeasible computationally. This emphasizes the imperative role of software fault tolerance (SFT) for autonomous applications. Several SFT techniques have been proposed, but failures revealed in VT applications imply that existing SFT techniques need to be fine-tuned. In this article, current replication-based SFT techniques are analyzed and classified with respect to their diversity, adjudication, and adaptivity. Essential parameters (reliability, time, variance, etc.) for adjudication, diversity, and adaptiveness are recorded. The identified parameters are mapped to different techniques (e.g., AFTRC, SCOP, VFT) for observing their shortcomings. Consequently, a generic framework named Diverse Parallel Adjudication for Software Fault Tolerance (DPA-SFT) is proposed. DPA-SFT addresses the shortcomings of existing SFT techniques for VTs with the added value of parallel and diverse adjudication. A prototype implementation of the proposed framework has been developed for assessing the viability of DPA-SFT over modules of VT. An empirical comparison of the proposed framework is performed with prevalent techniques (AFTRC, SCOP, VFT, etc). A thorough evaluation suggests that DPA-SFT performs better than contemporary SFT techniques in VTs due to its parallel and diverse adjudication.

Performance Comparison of Retry and N-Copy Software Fault Tolerance Techniques

Performance Comparison of Retry and N-Copy Software Fault Tolerance Techniques

FT-PBLAS: PBLAS-Based Fault-Tolerant Linear Algebra Computation on High-performance Computing Systems

As high-performance computing (HPC) systems have scaled up, resilience has become a great challenge. To guarantee resilience, various kinds of hardware and software techniques have been proposed. However, among popular software fault-tolerant techniques, both the checkpoint-restart approach and the replication technique face challenges of scalability in the era of peta- and exa-scale systems due to their numerous processes. In this situation, algorithm-based approaches, or algorithm-based fault tolerance (ABFT) mechanisms, have become attractive because they are efficient and lightweight. Although the ABFT technique is algorithm-dependent, it is possible to implement it at a low level (e.g., in libraries for basic numerical algorithms) and make it application-independent. However, previous ABFT approaches have mainly aimed at achieving fault tolerance in integrated circuits (ICs) or at the architecture level and are therefore not suitable for HPC systems; e.g., they use checksums of rows and columns of matrices rather than checksums of blocks to detect errors. Furthermore, they cannot deal with errors caused by node failure, which are common in current HPC systems. To solve these problems, this paper proposes FT-PBLAS, a PBLAS-based library for fault-tolerant parallel linear algebra computations that can be regarded as a fault-tolerant version of the parallel basic linear algebra subprograms (PBLAS), because it provides a series of fault-tolerant versions of interfaces in PBLAS. To support the underlying error detection and recovery mechanisms in the library, we propose a block-checksum approach for non-fatal errors and a scheme for addressing node failure, respectively. We evaluate two fault-tolerant mechanisms and FT-PBLAS on HPC systems, and the experimental results demonstrate the performance of our library.

Fault-tolerant real-time tasks scheduling with dynamic fault handling

Abstract Predictable performance when coping with transient failures is of paramount importance in safety-critical real-time systems. Various software fault-tolerant techniques are employed towards this goal among which check-pointing is a relatively cost-effective scheme. In this paper, we propose an efficient fault-tolerant scheduling framework with run-time fault handling protocol, where criticality levels can be adaptively inserted for fault handling according to run-time fault workload. In contrast to prior works which apply with task re-execution strategy, the proposed framework adaptively determines on-demand re-executions only on the faulty checkpoint segments, rather than on the whole job. Towards this, a unified overrun handling protocol is developed to handle fault recovery adaptively to avoid over-provisioning of resources. In addition, we develop an off-line schedulability analysis technique for the proposed scheduling algorithm. The simulation results show that our fault-tolerant scheduling framework can bring up to 81% improvement in supporting low-criticality service without sacrifice in the MC-schedulability compared with the existing techniques.

Error Detection by Diverse Instructions and Loop Optimization in DSP

For the digital signal processors (DSP), a new approach of detecting soft errors is proposed to overcome the transient fault, named EDIO. The goal of EDIO is to enhance the reliability of a DSP software system with reduced performance overhead, compared to former software fault tolerant techniques. EDIO employs diverse instructions for the sake of furthest exploring errors and increasing performance. The loop optimization delays the fault tolerance latency between errors detecting and errors handling, only in this way can EDIO schedule the software pipeline to reduce performance overhead significantly. We evaluate EDIO by the performance experiments and the ion irradiation experiments, which demonstrate exceptional fault coverage with a reasonable performance cost. Compared to the well-known software approach, EDIO demonstrates 6.4 times average speedup while remaining the ability to detect faults.

Designing fault-tolerant SOA based on design diversity

Over recent years, software developers have been evaluating the benefits of both Service-Oriented Architecture (SOA) and software fault tolerance techniques based on design diversity. This is achieved by creating fault-tolerant composite services that leverage functionally-equivalent services. Three major design issues need to be considered while building software fault-tolerant architectures based on design diversity: (i) selection of variants; (ii) selection of an adjudication algorithm to choose one of the results; and (iii) execution of variants. In addition, applications based on SOA need to function effectively in a dynamic environment where it is necessary to postpone decisions until runtime. In this scenario, control is highly distributed and involves conflicting user requirements. We aim to support the software architect in the design of fault-tolerant compositions. Leveraging a taxonomy for fault-tolerant systems, this paper proposes guidelines to aid software architects in making key design decisions. The taxonomy is used as the basis for defining a set of guidelines to support the architect in making decisions related to fault tolerance in SOA. The same taxonomy is used in a systematic literature review of solutions for fault-tolerant composite services. The review investigates how existing approaches for fault-tolerant composite services address design diversity issues and also specific issues related to SOA. The contribution of this work is twofold: (i) a set of guidelines for supporting the design of fault-tolerant SOA, based on a taxonomy for fault tolerance techniques; and (ii) a systematic literature review of existing solutions for designing fault-tolerant compositions using design diversity. Although existing solutions have made significant contributions to the development of fault-tolerant SOAs, there is a lack of approaches for fault-tolerant service composition that support strategies with diverse quality requirements and encompassing sophisticated context-aware capabilities. This paper discusses which design issues have been addressed by existing diversity-based approaches for fault-tolerant composite services. Finally, practical issues and difficulties are summarized and directions for future work are suggested.

Software health management

Software health management is defined as a technology that applies the principles and techniques of system health management to software systems. It is motivated by the apparent gap between the importance and complexity of software in today’s cyber-physical systems and the rare, but undoubtedly present occurrence of software malfunctions in those systems. While engineers strive to create dependable systems unforeseen environmental conditions or faults in the hardware can trigger latent defects in the software with potentially negative consequences. The goal of software health management is to maintain system function and performance, even when software fails in unexpected ways. System health management is a well-established discipline in aerospace systems: many air and space vehicles today have quite elaborate health management systems on board. Software fault tolerance techniques are also well-known and practiced since the days when computers were first used in critical applications. Software health management combines these directions: it borrows techniques like anomaly detection, fault diagnostics and mitigation from the first and techniques like triple modular redundancy and checkpoints and restarts from the second to manage the ‘health’ of the software system to maintain functionality and performance. Like in system health management, the goal of software health management is to prevent a software fault from becoming a system failure.

Deliberative, search-based mitigation strategies for model-based software health management

Rising software complexity in aerospace systems makes them very difficult to analyze and prepare for all possible fault scenarios at design time; therefore, classical run-time fault tolerance techniques such as self-checking pairs and triple modular redundancy are used. However, several recent incidents have made it clear that existing software fault tolerance techniques alone are not sufficient. To improve system dependability, simpler, yet formally specified and verified run-time monitoring, diagnosis, and fault mitigation capabilities are needed. Such architectures are already in use for managing the health of vehicles and systems. Software health management is the application of these techniques to software systems. In this paper, we briefly describe the software health management techniques and architecture developed by our research group. The foundation of the architecture is a real-time component framework (built upon ARINC-653 platform services) that defines a model of computation for software components. Dedicated architectural elements: the Component Level Health Manager (CLHM) and System Level Health Manager (SLHM) provide the health management services: anomaly detection, fault source isolation, and fault mitigation. The SLHM includes a diagnosis engine that (1) uses a Timed Failure Propagation Graph (TFPG) model derived from the component assembly model, (2) reasons about cascading fault effects in the system, and (3) isolates the fault source component(s). Thereafter, the appropriate system-level mitigation action is taken. The main focus of this article is the description of the fault mitigation architecture that uses goal-based deliberative reasoning to determine the best mitigation actions for recovering the system from the identified failure mode.

A Dependent Model for Fault Tolerant Software Systems During Debugging

This paper proposes a special redundant model for describing the s-dependency of multi-version programming software during testing and debugging. N-version programming (NVP) is one of the most important software fault tolerance techniques. Many papers have studied the issue of fault correlation among versions. However, only a few of them consider this issue during the testing and debugging part of the software development life cycle. During testing and debugging, faults may not be successfully removed. Imperfect debugging may result in unsuccessful removal, and the introduction of new faults. Different from existing NVP models, the model proposed in this paper allows an assessment of s-dependency when correlated failures may not necessarily occur at the same execution time point. The model focuses on 2 VP systems. It is developed to be a bivariate counting process by assuming positive s-dependency among versions. Considering imperfect debugging, this bivariate process characterizes dynamic changes of fault contents for each version during testing and debugging. The system reliability, expected number of faults, probability of perfect debugging, and parameter estimation of model parameters are presented. An application example is given to illustrate the proposed model. The paper provides an alternative approach for evaluating the reliability of 2 VP software systems when there is positive s -dependency between versions.

Testing software fault tolerance techniques

Software fault tolerance can itself be dangerous error-prone because of the additional effort that must be included in the programming process. The paper is based upon research in the area of testing software fault tolerance techniques. A Framework to Test Fault Tolerance has been proposed and validated with industry data. During the validation process we identified some of the interesting findings that can be explored to carry out further research in this area.

Optimal Fault Tolerance Strategy Selection for Web Services

Service-oriented systems are usually composed by heterogeneous Web services, which are distributed across the Internet and provided by organizations. Building highly reliable service-oriented systems is a challenge due to the highly dynamic nature of Web services. In this paper, the authors apply software fault tolerance techniques for Web services, where the component failures are handled by fault tolerance strategies. In this paper, a distributed fault tolerance strategy evaluation and selection framework is proposed based on versatile fault tolerance techniques. The authors provide a systematic comparison of various fault tolerance strategies by theoretical formulas, as well as real-world experiments. This paper also presents the optimal fault tolerance strategy selection algorithm, which employs both the QoS performance of Web services and the requirements of service users for selecting optimal fault tolerance strategy. A prototype is implemented and real-world experiments are conducted to illustrate the advantages of the evaluation framework. In these experiments, users from six different locations perform evaluation of Web services distributed in six countries, where over 1,000,000 test cases are executed in a collaborative manner to demonstrate the effectiveness of this approach.

Low-cost error containment and recovery for onboard guarded software upgrading and beyond

Message-driven (MDCD) error containment and recovery, a low-cost approach to mitigating the effect of software design faults in distributed embedded systems, is developed for onboard guarded software upgrading for deep-space missions. In this paper, we first describe and verify the MDCD algorithms in which we introduce the notion of confidence-driven to complement the communication-induced approach employed by a number of existing checkpointing protocols to achieve error containment and recovery efficiency. We then conduct a model-based analysis to show that the algorithms ensure low performance overhead. Finally, we discuss the advantages of the MDCD approach and its potential utility as a general-purpose, low-cost software fault tolerance technique for distributed embedded computing.

Analysis of Checkpointing for Real-Time Systems

Predictable performance in the event of failures is of paramount importance in most safety critical real-time systems. Various hardware as well as software fault-tolerant techniques are employed to...

Supporting customized failure models for distributed software

The cost of employing software fault tolerance techniques in distributed systems is strongly related to the type of failures to be tolerated. For example, in terms of the amount of redundancy required and execution time, tolerating a processor crash is much cheaper than tolerating arbitrary (or Byzantine) failures. This paper describes an approach to constructing configurable services for distributed systems that allows easy customization of the type of failures to tolerate. Using this approach, it is possible to configure custom services across a spectrum of possibilities, from a very efficient but unreliable server group that does not tolerate any failures, to a less efficient but reliable group that tolerates crash, omission, timing, or arbitrary failures. The approach is based on building configurable services as collections of software modules called micro-protocols. Each micro-protocol implements a different semantic property or property variant, and interacts with other micro-protocols using an event-driven model provided by a runtime system. In addition to facilitating the choice of failure model, the approach allows service properties such as message ordering and delivery atomicity to be customized for each application.

SACI_FTR_LOADER: A Dynamic Loader for the On-Board Computer of the First Brazilian Microssatelite

This paper describes the conception and implementation of a dynamic loader, called SACI_FTR_LOADER, which will be used to upload programs from a ground station onto the onboard computer (called Trisputer) of the first Brazilian microssatelite. The satellite, named SACI (Satellite for Scientific Applications), will be launched by the middle of the next year. The loader is part of the real-time operating system designed to control the resources of the machine, which also incorporates software fault tolerance techniques. The Trisputer is a fault-tolerant multiprocessor computer system which was designed having in mind the high reliability requirements associated with the mission. This goal was achieved by introducing fault tolerance at both hardware and software levels. The on-board computer consists of three processing modules, each one based on a Transputer T-805-30 INMOS microprocessor

Analysis of preventive maintenance in transactions based software systems

Preventive maintenance of operational software systems, a novel technique for software fault tolerance, is used specifically to counteract the phenomenon of software "aging". However, it incurs some overhead. The necessity to do preventive maintenance, not only in general purpose software systems of mass use, but also in safety-critical and highly available systems, clearly indicates the need to follow an analysis based approach to determine the optimal times to perform preventive maintenance. In this paper, we present an analytical model of a software system which serves transactions. Due to aging, not only the service rate of the software decreases with time, but also the software itself experiences crash/hang failures which result in its unavailability. Two policies for preventive maintenance are modeled and expressions for resulting steady state availability, probability that an arriving transaction is lost and an upper bound on the expected response time of a transition are derived. Numerical examples are presented to illustrate the applicability of the models.

Implementing atomic actions in Ada 95

Atomic actions are an important dynamic structuring technique that aid the construction of fault-tolerant concurrent systems. Although they were developed some years ago, none of the well-known commercially-available programming languages directly support their use. This paper summarizes software fault tolerance techniques for concurrent systems, evaluates the Ada 95 programming language from the perspective of its support for software fault tolerance, and shows how Ada 95 can be used to implement software fault tolerance techniques. In particular, it shows how packages, protected objects, requeue, exceptions, asynchronous transfer of control, tagged types, and controlled types can be used as building blocks from which to construct atomic actions with forward and backward error recovery, which are resilient to deserter tasks and task abortion.

Methodology for cost-effective software fault tolerance for mission-critical systems

A computing capabilities continue to advance, there will be a concurrent rise in the number of both hardware and software faults. These will be caused by the greater volume of more complex software, by the increased number of untested software states, and by more incidents of hardware/software interaction faults as a result of increased hardware speed and density. The traditional software implemented fault tolerance: approaches have been successfully utilized in life-critical systems, such as digital flight controls, where their additional costs can be easily justified. Examples include N-Version Programming and Recovery Block approaches. However, there is still a need for dependable computing for mission-critical applications as well. Often, these traditional techniques are avoided for mission-critical systems due to the difficulty in justifying their extra up-front development cost. We provide an alternative for the high "sunk cost" of traditional software fault tolerance techniques. The methodology, called data fusion integrity processes (DFIPs), is a simple, yet effective technique for mission-critical systems. In addition, the approach establishes a framework from which other costlier, more extensive traditional techniques can be added. We present details of the DFIP methodology and a DFIP framework for Ada programs. We also briefly discuss development of a DFIP code generation system which exploits Java that will enable users to quickly build a DFIP framework in Ada, and select reusable DFIP component methods.

Coupling algorithms for replicated objects and processes

The problem of defining a general scheme for interfacing replication for software fault tolerance (diverse software versions) and system replication techniques (distributed execution of process replicas) is investigated. An algorithm, called Minimum Information Algorithm, is defined in order to couple object copies and process replicas.