Software-implemented Fault Injection Research Articles

Improving the Accuracy of IR-Level Fault Injection

Fault injection (FI) is a commonly used experimental technique to evaluate the resilience of software techniques for tolerating hardware faults. Software-implemented FI can be performed at different levels of abstraction in the system stack; FI performed at the compiler&#x2019;s intermediate representation (IR) level has the advantage that it is closer to the program being evaluated and is hence easier to derive insights from for the design of software fault-tolerance mechanisms. Unfortunately, it is not clear how accurate IR-level FI is vis-a-vis FI performed at the assembly code level, and prior work has presented contradictory findings. In this article, we perform a comprehensive evaluation of the accuracy of IR-level FI across a range of benchmark programs and compiler optimization levels. Our results show that IR-level FI is as accurate as assembly-level FI for silent data corruption (SDC) probability estimation across different benchmarks and optimization levels. Further, we present a machine-learning-based technique for improving the accuracy of <i>crash</i> probability measurements made by IR-level FI, which takes advantage of an observed correlation between program crash probabilities and instructions that operate on memory address values. We find that the machine learning technique provides comparable accuracy for IR-level FI as assembly code level FI for program crashes.

Using Fault Injection to Assess Blockchain Systems in Presence of Faulty Smart Contracts

Blockchain has become particularly popular due to its promise to support business-critical services in very different domains (e.g., retail, supply chains, healthcare). Blockchain systems rely on complex middleware, like Ethereum or Hyperledger Fabric, that allow running smart contracts, which specify business logic in cooperative applications. The presence of software defects or faults in these contracts has notably been the cause of failures, including severe security problems. In this paper, we use a software implemented fault injection (SWIFI) technique to assess the behavior of permissioned blockchain systems in the presence of faulty smart contracts. We emulate the occurrence of general software faults (e.g., missing variable initialization) and also blockchain-specific software faults (e.g., missing require statement on transaction sender) in smart contracts code to observe the impact on the overall system dependability (i.e., reliability and integrity). We also study the effectiveness of formal verification (i.e., done by solc-verify) and runtime protections (e.g., using the assert statement) mechanisms in detection of injected faults. Results indicate that formal verification as well as additional runtime protections have to complement built-in platform checks to guarantee the proper dependability of blockchain systems and applications. The work presented in this paper allows smart contract developers to become aware of possible faults in smart contracts and to understand the impact of their presence. It also provides valuable information for middleware developers to improve the behavior (e.g., overall fault tolerance) of their systems.

Evaluation of effectiveness of fault-tolerant techniques in a digital instrumentation and control system with a fault injection experiment

Abstract Recently, instrumentation and control (I&C) systems in nuclear power plants have undergone digitalization. Owing to the unique characteristics of digital I&C systems, the reliability analysis of digital systems has become an important element of probabilistic safety assessment (PSA). In a reliability analysis of digital systems, fault-tolerant techniques and their effectiveness must be considered. A fault injection experiment was performed on a safety-critical digital I&C system developed for nuclear power plants to evaluate the effectiveness of fault-tolerant techniques implemented in the target system. A software-implemented fault injection in which faults were injected into the memory area was used based on the assumption that all faults in the target system will be reflected in the faults in the memory. To reduce the number of required fault injection experiments, the memory assigned to the target software was analyzed. In addition, to observe the effect of the fault detection coverage of fault-tolerant techniques, a PSA model was developed. The analysis of the experimental result also can be used to identify weak points of fault-tolerant techniques for capability improvement of fault-tolerant techniques.

NMR-MPar: A Fault-Tolerance Approach for Multi-Core and Many-Core Processors

Multi-core and many-core processors are a promising solution to achieve high performance by maintaining a lower power consumption. However, the degree of miniaturization makes them more sensitive to soft-errors. To improve the system reliability, this work proposes a fault-tolerance approach based on redundancy and partitioning principles called N-Modular Redundancy and M-Partitions (NMR-MPar). By combining both principles, this approach allows multi-/many-core processors to perform critical functions in mixed-criticality systems. Benefiting from the capabilities of these devices, NMR-MPar creates different partitions that perform independent functions. For critical functions, it is proposed that N partitions with the same configuration participate of an N-modular redundancy system. In order to validate the approach, a case study is implemented on the KALRAY Multi-Purpose Processing Array (MPPA)-256 many-core processor running two parallel benchmark applications. The traveling salesman problem and matrix multiplication applications were selected to test different device’s resources. The effectiveness of NMR-MPar is assessed by software-implemented fault-injection. For evaluation purposes, it is considered that the system is intended to be used in avionics. Results show the improvement of the application reliability by two orders of magnitude when implementing NMR-MPar on the system. Finally, this work opens the possibility to use massive parallelism for dependable applications in embedded systems.

A Light-Weight Fault Injection Approach to Test Automated Production System PLC Software in Industrial Practice

A light-weight software-implemented fault injection (SWIFI) testing approach is introduced, focusing on technical process faults and system faults. The reaction of automated production systems (aPSs) and their programmable logic controller (PLC) software to these faults is tested. In order to tailor the testing approach to the aPS domain in industrial practice, our test generation is based on a classification of possible deviations, i.e. a classification of possible technical process and system faults as the PLC perceives them. As a result, both specification and test execution become more efficient for practitioners. Furthermore, the test specification is tailored for execution on IEC 61131-3 programming environments. In this, the execution of test cases both against simulation or the real aPS, is enabled.

A Software-Implemented Fault Injection Toolkit for Dependency Analysis of Large Scale Distributed Applications

A Software-Implemented Fault Injection Toolkit for Dependency Analysis of Large Scale Distributed Applications

XML schema based fault set definition to improve fault injection tools interoperability

Software implemented fault injection tools (SWIFI) use fault injectors to carry out the fault injection campaign defined in a GUI-based application. However, the communication between the fault injector and the application is defined in an ad-hoc manner. This paper describes an XML schema formalisation approach for the definition of fault sets which specify low level memory and/or register value corruptions in embedded microprocessor-based systems and resource usage faults in host based systems. Through this proposed XML schema definition, different injectors could be used to carry out the same fault set injection. To validate this approach an experimental tool called Exhaustif®, consisting of a GUI Java application for defining the fault sets and injection policies, one injector for Windows hosts systems and two injectors for Sparc and i386 architectures under RTEMS have been developed.

Error Detection Enhancement in PowerPC Architecture-based Embedded Processors

This paper presents a behavior-based error detection technique called Control Flow Checking using Branch Trace Exceptions for PowerPC processors family (CFCBTE). This technique is based on the branch trace exception feature available in the PowerPC processors family for debugging purposes. This technique traces the target addresses of program branches at run-time and compares them with reference target addresses to detect possible violations caused by transient faults. The reference target addresses are derived by a preprocessor from the source program. To enhance the error detection coverage, three other mechanisms, i.e., Machine Check Exception, System Trap Instructions and Work Load Timer are combined with the Branch Trace Exception mechanism. The proposed technique is experimentally evaluated on a 32-bit PowerPC microcontroller using software implemented fault injection (SWIFI) and Power Supply Disturbances fault injection (PSD). A total of 6,000 faults were injected in microcontroller to measure the error detection coverage of the proposed control flow checking technique. The experimental results show that this technique detects about 95.2% of transient errors in software implemented fault injection method and 96.4% of transient errors in power supply disturbances fault injection method.

A Conformance Testing Process for Space Applications Software Services

Comprehensive tests for space applications software are costly but extremely necessary. These software must be reliable and produced within schedule and budget. In a tentative of make the space mission software development for space agencies and industries more costeffective, the European Committee for Space Standardization (ECSS) has been compiling a set of standards that specify the common core of some space application systems. Once the set of services is standardized, the conformance problem is raised. In this paper we present a testing process for standardized services, which is based on the IS-9646 standard for ISO protocol conformance testing. The process includes an approach to derive test and fault cases by combining conformance testing concepts with the software-implemented fault injection (SWIFI) technique. One advantage of this process is the generation of a re-usable

Embedded system dependability evaluation with simulation tools

Dependability evaluation is an important issue in designing and exploiting embedded systems. The paper presents the capabilities and limitations of fault injection techniques in relevance to dependability analysis. e discuss the impact of fault models on the simulation results and their interpretation. A special attention is paid to transient faults, which dominate in contemporary systems. The presented analysis bases mostly on our experience with software implemented fault injection. As an illustration we give some original experimental results for a real application.

Web Service Based Software Implemented Fault Injection

Web Service Based Software Implemented Fault Injection

Error Detection Enhancement in COTS Superscalar Processors with Performance Monitoring Features

Increasing use of commercial off-the-shelf (COTS) superscalar processors in industrial, embedded, and real-time systems necessitates the development of error detection mechanisms for such systems. This shows an error detection scheme called committed instructions counting (CIC) to increase error detection in such systems. The scheme uses internal performance monitoring features and an external watchdog processor (WDP). The performance monitoring features enable counting the number of committed instructions in a program. The scheme is experimentally evaluated on a 32-bit Pentium/spl reg/ processor using software implemented fault injection (SWIFI). A total of 8181 errors were injected into the Pentium/spl reg/ processor. The results show that the error detection coverage varies between to 90.92% and 98.41%, for different workloads.

Empaquetâches de tolérance aux fautes pour les systèmes temps réel

The integration of software COTS components into safety-critical real-time systems raises serious problems from a dependability viewpoint. Wrappers can be used to assist such an integration by providing additional error detection mechanisms. We recently defined a framework to automatically develop such a type of wrappers from temporal logic specifications. In this paper, we extend this framework to endow wrappers with forward error recovery capabilities based on the concept of recovery actions. Recovery actions consist of variants of basic functions of the target system that are run concurrently to help correct errors. We apply these comprehensive wrappers to a real-time COTS microkernel. Software implemented fault injection (SWIFI) is then used to evaluate the efficiency of the wrappers. This step helped identify the sets of wrappers that maximize the error detection and recovery coverage while meeting hard task deadlines.

Comparison of physical and software-implemented fault injection techniques

This paper addresses the issue of characterizing the respective impact of fault injection techniques. Three physical techniques and one software-implemented technique that have been used to assess the fault tolerance features of the MARS fault-tolerant distributed real-time system are compared and analyzed. After a short summary of the fault tolerance features of the MARS architecture and especially of the error detection mechanisms that were used to compare the erroneous behaviors induced by the fault injection techniques considered, we describe the common distributed testbed and test scenario implemented to perform a coherent set of fault injection campaigns. The main features of the four fault injection techniques considered are then briefly described and the results obtained are finally presented and discussed. Emphasis is put on the analysis of the specific impact and merit of each injection technique.

EXFI

Evaluating the faulty behavior of low-cost embedded microprocessor-based boards is an increasingly important issue, due to their adoption in many safety critical systems. The architecture of a complete Fault Injection environment is proposed, integrating a module for generating a collapsed list of faults, and another for performing their injection and gathering the results. To address this issue, the paper describes a software-implemented Fault Injection approach based on the Trace Exception Mode available in most microprocessors. The authors describe EXFI, a prototypical system implementing the approach, and provide data about some sample benchmark applications. The main advantages of EXFI are the low cost, the good portability, and the high efficiency

Dependability analysis of a high-speed network using software-implemented fault injection and simulated fault injection

This paper presents a dependability study of high-speed, switched Local Area Networks (LANs) using Myrinet as an example testbed (with theoretical speeds of 2.56 Gbps). The study uses results of two fault injection methods, simulated fault injection and software-implemented fault injection (SWIFI), to analyze the application-level impact of transient faults injected into the network interface hardware. These results include a number of errors, such as dropped or corrupt messages, host interface or host resets, and local or remote host interface hangs. The paper presents the study in two parts: First, the results from the SWIFI method in the real system are used as a basis to validate the simulation and identify the major factors leading to differences between the methods. A comparison between the two injection methods shows that they agree for 83 percent of the fault injections. The results, however, vary greatly, depending on the fault type considered. The study also presents an analysis of the effects of varying workload intensity, host platform, and interface function targeted by the injection. An example of this analysis is to show that the function targeted has a significant impact on the fault activation rate. Finally, the study identifies two mechanisms by which faults may propagate from the interface to other parts of the network; in one example, this propagation caused the interface's host computer to reboot, while another caused a remote interface in the network to hang.

Xception: a technique for the experimental evaluation of dependability in modern computers

An important step in the development of dependable systems is the validation of their fault tolerance properties. Fault injection has been widely used for this purpose, however with the rapid increase in processor complexity, traditional techniques are also increasingly more difficult to apply. This paper presents a new software-implemented fault injection and monitoring environment, called Xception, which is targeted at modern and complex processors. Xception uses the advanced debugging and performance monitoring features existing in most modern processors to inject quite realistic faults by software, and to monitor the activation of the faults and their impact on the target system behavior in detail. Faults are injected with minimum interference with the target application. The target application is not modified, no software traps are inserted, and it is not necessary to execute the target application in special trace mode (the application is executed at full speed). Xception provides a comprehensive set of fault triggers, including spatial and temporal fault triggers, and triggers related to the manipulation of data in memory. Faults injected by Xception can affect any process running on the target system (including the kernel), and it is possible to inject faults in applications for which the source code is not available. Experimental, results are presented to demonstrate the accuracy and potential of Xception in the evaluation of the dependability properties of the complex computer systems available nowadays.