Fault-tolerant Implementation Research Articles

Analysis for performance and reliability of fault-tolerant parallel software

We propose a technique for constructing a fault-tolerant parallel software for general commercial massively parallel computers which are not provided with special fault-tolerant functions. This technique is a hybrid of the primary/backup approach and state machine approach, and can implement parallel programs in fault tolerance by automatically converting user programs. In general, when a parallel system is to be used as a fault-tolerant computer, since parallel entities are used as redundant elements for obtaining fault tolerance, the maximum performance will decrease concurrently with the improvement of reliability. Moreover, it is necessary to consider the performance drop for processing which is supplementary to the original program in fault-tolerant implementation by software. Therefore, a gain by fault-tolerant implementation cannot be shown if it is merely demonstrated that an improvement of the reliability is obtained. In this paper, we define an evaluation index which takes into account reliability improvement and performance drop; based on this index, we study the execution environment which can tolerate practical use for fault-tolerant parallel software. © 2000 Scripta Technica Syst Comp Jpn, 31(7): 56–65, 2000

Joint sensor fault detection for fault tolerant parallel manipulators

Parallel manipulators with redundant joint displacement sensing can be exploited to develop fault tolerant implementations. This is possible since fundamental problems of the associated kinematics can still be solved after the elimination of faulty sensor readings. The ability of detecting faulty sensor readings is a requirement of any fault tolerant implementation scheme. A sensor fault detection method is presented for redundantly sensed parallel manipulators. A broad class of three-branch manipulators is considered where each branch consists of three main-arm joints and supports a common payload through respective passive spherical joints. The detection method is based on the comparison of forward displacement solutions for different cases of joint sensor readings. The existence of common solutions based on the branches–sensors considered, is used to effectively identify the existence of a failed sensor. Once a faulty sensor is identified, continued (fault tolerant) operation is possible using a forward displacement solution based on the readings of the accurate sensors. The detection method is implemented in a computer simulation of a calibrated three-branch parallel manipulator. © 2000 John Wiley & Sons, Inc.

Simulation-based Testing of Communication Protocols for Dependable Embedded Systems

We present a novel approach to testing fault-tolerant and real-time protocol implementations. Cesium, our testing environment, executes the protocols in a centralized simulator of the distributed s...

Hierarchical error detection in a software implemented fault tolerance (SIFT) environment

Proposes a hierarchical error detection framework for a software-implemented fault tolerance (SIFT) layer of a distributed system. A four-level error detection hierarchy is proposed in the context of Chameleon, a software environment for providing adaptive fault tolerance in an environment of commercial off-the-shelf (COTS) system components and software. The design and implementation of a software-based distributed signature monitoring scheme, which is central to the proposed four-level hierarchy, is described. Both intra-level and inter-level optimizations that minimize the overhead of detection and are capable of adapting to runtime requirements are proposed. The paper presents results from a prototype implementation of two levels of the error detection hierarchy and results of a detailed simulation of the overall environment. The results indicate a substantial increase in availability due to the detection framework and help in understanding the tradeoffs between overhead and coverage for different combinations of techniques.

Good quantum-convolutional error-correction codes and their decoding algorithm exist

Quantum convolutional code was introduced recently as an alternative way to protect vital quantum information. To complete the analysis of quantum convolutional code, I report a way to decode certain quantum convolutional codes based on the classical Viterbi decoding algorithm. This decoding algorithm is optimal for a memoryless channel. I also report three simple criteria to test if decoding errors in a quantum convolutional code will terminate after a finite number of decoding steps whenever the Hilbert space dimension of each quantum register is a prime power. Finally, I show that certain quantum convolutional codes are in fact stabilizer codes. And hence, these quantum stabilizer convolutional codes have fault-tolerant implementations.

Fault-tolerant neural networks with higher functionality

It is known that mutually coupled neural networks are suited for solving optimization problems. So far, we have shown that a neural network which is tolerant to unidirectional faults can be realized by selecting automatically one of the two complementary representations of the solution. It is also shown that a fault-tolerant neural network can be realized by a triplication structure with an appropriate merging function, even if both stuck-at-1 and 0 faults may occur simultaneously. In these realizations, the conventional property of a neuron, a component of the neural network, is utilized, where the output of the neuron is determined by the sigmoid function of the sum of its inputs multiplied by the respective weights (connection coefficients). This paper shows that if a neuron can have a higher function than the weighted sum of its inputs, a neural network which is tolerant to stuck-at-1 and 0 mixed faults can be realized by the duplication structure. Since the duplication structure suffices, it is expected that less hardware will be needed than in the triplication structure. Although the higher function of the neuron assumed in this paper has not actually been realized, we expect that such a function will be realized in hardware in the near future as a result of progress in integrated circuit technology. In this sense, the proposal in this paper is practically realistic, and provides a new means of implementing fault tolerance of neural networks. © 1999 Scripta Technica, Syst Comp Jpn, 30(10): 22–33, 1999

Advanced fault tolerant computing for future manned space missions

Planet Mars has been at the focal point of astronomical attention. As we enter the next millennium, Mars will play a key role in humanity's successful expansion into heliocentric space. Future Mars space transportation will require reliable operations over a lifespan of years. Unlike other long life deep space missions, human Mars operations require real-time masking of critical faults. This paper discusses Mars stringent baseline requirements and constraints and presents fault-tolerant approaches, techniques, and design building block strategies that include standby redundancy, reconfigurable voting, backup sparing, and graceful degradation. The contemporary approach and recognized inadequacies of their application to long duration space missions will be discussed. Certain problems will be identified and viable solutions offered. Various aspects of fault tolerant designs and implementations are discussed including component selection, radiation tolerance, high-density packaging technology, computational integrity, and fault coverage. Architectural solutions that can make systems affordable such as open systems, standardization, and ease of validation will be highlighted. The paper concludes with a technology demonstration plan to achieve the desired baseline requirements and goals.

Fault‐tolerant RT‐Mach (FT‐RT‐Mach) and an application to real‐time train control

Even though real-time systems have the stringent constraint of completing tasks before their deadlines, many existing real-time operating systems do not implement fault tolerance capabilities. In this paper we summarize fault tolerant real-time scheduling policy for dynamic tasks with ready times and deadlines. Our focus in this paper is the implementation, which includes fault-tolerant scheduling, re-scheduling, and recovery mechanisms in the FT-RT-Mach operating system, a fault-tolerant version of RT-Mach. A real-time train control application is then implemented using the FT-RT-Mach operating system. Copyright © 1999 John Wiley & Sons, Ltd.

Fault-tolerant RT-Mach (FT-RT-Mach) and an application to real-time train control

Even though real-time systems have the stringent constraint of completing tasks before their deadlines, many existing real-time operating systems do not implement fault tolerance capabilities. In this paper we summarize fault tolerant real-time scheduling policy for dynamic tasks with ready times and deadlines. Our focus in this paper is the implementation, which includes fault-tolerant scheduling, re-scheduling, and recovery mechanisms in the FT-RT-Mach operating system, a fault-tolerant version of RT-Mach. A real-time train control application is then implemented using the FT-RT-Mach operating system. Copyright © 1999 John Wiley & Sons, Ltd.

The Instancy of Snapshots and Commuting Objects

We present a sequence of constructions of commuting synchronization objects (e.g.,fetch-and-incrementandfetch-and-add) in a system ofnprocessors from any two processor synchronization objects whose consensus number is two or more (Herlihy, “Proceedings of the Tenth ACM Symposium on Principles of Distributed Computing, 1991,” pp. 11–22). Each implementation in the sequence uses a particular type of shared memory snapshot as a building block. Later implementations in the sequence are based on higher quality snapshots. The first implementation of afetch-and-incrementuses the standard atomic snapshot concept, introduced by Afek et al. and Anderson (Afek et al.,J. Assoc. Comput. Mach.40(4) (1993), 873–890; Anderson, “Proceedings of the Ninth Annual Assoc. Comput. Mach. Symposium on Principles of Distributed Computing, August 1990,” pp. 15–29) while the last construction in the sequence, offetch-and-add, is based on the immediate snapshot concept introduced in (Borowsky and Gafni, “Proceedings of the 12th Assoc. Comput. Mach. Symposium on Principles of Distributed Computing, August 1993,” pp. 41–51). This last construction also yields an implementation of a stronger snapshot, which we callWrite-and-snapshot. In addition, this work solves an open question of Borowsky and Gafni by presenting an implementation of a multishot immediate snapshot object. Additional implications of our constructions are (1) the existence of fault-tolerant self implementations of commuting objects, (2) improvements in the efficiency of randomized constructions of commuting objects from read/write registers, and (3) low contention constructions of commuting objects.

Fault-tolerant wait-free shared objects

Wait-free implementations of shared objects tolerate the failure of processes, but not the failure of base objects from which they are implemented. We consider the problem of implementing shared objects that tolerate the failure of both processes and base objects.We identify two classes of object failures:responsiveandnonresponsive. With responsive failures, a faulty object responds to every operation, but its responses may be incorrect. With nonresponsive failures, a faulty object may also “hang” without responding. In each class, we definecrash, omission,andarbitrarymodes of failure.We show that all responsive failure modes can be tolerated. More precisely, for all responsive failure modes ℱ, object typesT, andt≥ 0, we show how to implement a shared object of typeTwhich ist-tolerant for ℱ. Such an object remains correct and wait-free even if up totbase objects fail according to ℱ. In contrast to responsive failures, we show that even the most benign non-responsive failure mode cannot be tolerated. We also show that randomization can be used to circumvent this impossibility result.Graceful degradationis a desirable property of fault-tolerant implementations: the implemented object never fails more severely than the base objects it is derived from, even if all the base objects fail. For several failure modes, we show wheter this property can be achieved, and, if so, how.

Performance of fault-tolerant data and compute intensive programs over a network of workstations

A well-known structuring technique for a wide class of parallel applications is the bag of tasks, which allows a computation to be partitioned dynamically between a collection of concurrent processes. This paper describes a fault-tolerant implementation of this structure using atomic actions ( atomic transactions) to operate on persistent objects, which are accessed in a distributed setting via a Remote Procedure Call (RPC). The system developed is suited to parallel execution of data and compute intensive programs that require persistent storage and fault tolerance facilities. The suitability of the system is examined in the context of the measured performance of three specific applications; ray tracing, matrix multiplication and Cholesky factorisation. The measured performance compares well with that predicted by analysis of the applications based on benchmark measurement of hardware parameters. The same analysis allows modelling of hardware upgrades.

Error detection and diagnosis for fault tolerance in distributed systems

The early error detection and the understanding of the nature and conditions of an error occurrence can be useful to make an effective and efficient recovery in distributed systems. Various distributed system extensions were introduced for the implementation of fault tolerance in distributed software systems. These extensions rely mainly on the exchange of contextual information appended to every transmitted application specific message. Ideally, this information should be used for checkpointing, error detection, diagnosis and recovery should a transient failure occur later during the distributed program execution. In this paper, we present a generalized extension suitable for fault-tolerant distributed systems such as communication software systems and its detection capabilities are shown. Our extension is based on the execution of message validity test prior to the transmission of messages and the piggybacking of contextual information to facilitate the detection and diagnosis of transient faults in the distributed system.

Survey Of Backward Error Recovery Techniques For Multicomputers Based On Checkpointing And Rollback

Backward error recovery, baaed on checkpointing and rollback, is often used for implementing fault tolerance in multicomputer systems. During failure-free operation the process states are regularly saved, and after a fault is detected the system is rolled back to a previously saved state. Four classes of techniques can be distinguished: semiautomatic techniques, message logging, coordinated checkpointing, and hybrid techniques. The authors provide a survey of these alternatives and discuss the overhead possibly involved, allowing the user to choose an optimal checkpointing and rollback technique for given faciUties and applications.

Implementation of design fault tolerance based on a standardization project for operating systems

Many designers of high-reliability systems are now turning to design diversity as a technique of design fault tolerance. In this approach, parallel components in a system are not identical, but have different internal design and are developed by multiple teams working independently to a single external specification. Treating a standardization project involving multiple implementers as equivalent to design diversity, the authors have been investigating a multilevel design diversity architecture in which the design diversity approach is applied to application programs, the operating system, and hardware in accordance with the TRON standardization guidelines. This paper reports a backup error recovery mechanism for the operating system, focusing on a reconfiguration mechanism for modules that differ in their internal design. The mechanism was developed for modules developed independently by three suppliers under the ITRON2 operating system specification. The provision for reconfiguration required the rewriting of only 0.8% of the source code for the modules. © 1997 Scripta Technica, Inc. Syst Comp Jpn. 28 (1): 113–120, 1997

Methodology for cost-effective software fault tolerance for mission-critical systems

A computing capabilities continue to advance, there will be a concurrent rise in the number of both hardware and software faults. These will be caused by the greater volume of more complex software, by the increased number of untested software states, and by more incidents of hardware/software interaction faults as a result of increased hardware speed and density. The traditional software implemented fault tolerance: approaches have been successfully utilized in life-critical systems, such as digital flight controls, where their additional costs can be easily justified. Examples include N-Version Programming and Recovery Block approaches. However, there is still a need for dependable computing for mission-critical applications as well. Often, these traditional techniques are avoided for mission-critical systems due to the difficulty in justifying their extra up-front development cost. We provide an alternative for the high "sunk cost" of traditional software fault tolerance techniques. The methodology, called data fusion integrity processes (DFIPs), is a simple, yet effective technique for mission-critical systems. In addition, the approach establishes a framework from which other costlier, more extensive traditional techniques can be added. We present details of the DFIP methodology and a DFIP framework for Ada programs. We also briefly discuss development of a DFIP code generation system which exploits Java that will enable users to quickly build a DFIP framework in Ada, and select reusable DFIP component methods.

Task-configuration of a pearl-based programmable controller for process automation

This paper describes the configuration of real-time tasks using a PEARL-based programmable controller (“PEARL-PLC”). It will be shown that tasks can be consistently configured with function-blocks for logic, sequence and feedback control. Furthermore, the synchronization of cooperating tasks, controlling interdependent sub-processes, can also be configured using communication function-blocks based on Multicomputer-PEARL. Moreover, using the explicit task-scheduling offered by PEARL, the needed processor-time can be reduced to a conceptual minimum. Details of the configuration of a sequence-control task and a fuzzy-control task, the latter as a representative of a feedback-control task, are presented. Finally, trends of future developments of the “PEARL-PLC”, e.g. the implementation of fault-tolerance, are pointed out.

The Determinism Insurance in The Sceptre 2 Fault Tolerance Model

This paper presents some aspects of the implementation of fault tolerance in the Sceptre 2 real-time operating system. The accent is particularly put on the determinism insurance problem. The choosen approach is based on capsules replication on distinct nodes and the semi-active redundancy is used.

Focused fault injection testing of software implemented fault tolerance mechanisms of Voltan TMR nodes

One way of gaining confidence in the adequacy of fault tolerance mechanisms of a system is to test the system by injecting faults and see how the system performs under faulty conditions. This paper presents an application of the focused fault injection method that has been developed for testing software implemented fault tolerance mechanisms of distributed systems. The method exploits the object oriented approach of software implementation to support the injection of specific classes of faults. With the focused fault injection method, the system tester is able to inject specific classes of faults (including malicious ones) such that the fault tolerance mechanisms of a target system can be tested adequately. The method has been applied to test the design and implementation of voting, clock synchronization, and ordering modules of the Voltan TMR (triple modular redundant) node. The tests performed uncovered three flaws in the system software.

Formal verification for fault-tolerant architectures: prolegomena to the design of PVS

PVS is the most recent in a series of verification systems developed at SRI. Its design was strongly influenced, and later refined, by our experiences in developing formal specifications and mechanically checked verifications for the fault-tolerant architecture, algorithms, and implementations of a model "reliable computing platform" (RCP) for life-critical digital flight-control applications, and by a collaborative project to formally verify the design of a commercial avionics processor called AAMP5. Several of the formal specifications and verifications performed in support of RCP and AAMP5 are individually of considerable complexity and difficulty. But in order to contribute to the overall goal, it has often been necessary to modify completed verifications to accommodate changed assumptions or requirements, and people other than the original developer have often needed to understand, review, build on, modify, or extract part of an intricate verification. We outline the verifications performed, present the lessons learned, and describe some of the design decisions taken in PVS to better support these large, difficult, iterative, and collaborative verifications.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>