Fault Injection Experiments Research Articles

Automating Fault Test Cases Generation and Execution for Automotive Safety Validation via NLP and HIL Simulation

The complexity and the criticality of automotive electronic implanted systems are steadily advancing and that is especially the case for automotive software development. ISO 26262 describes requirements for the development process to confirm the safety of such complex systems. Among these requirements, fault injection is a reliable technique to assess the effectiveness of safety mechanisms and verify the correct implementation of the safety requirements. However, the method of injecting the fault in the system under test in many cases is still manual and depends on an expert, requiring a high level of knowledge of the system. In complex systems, it consumes time, is difficult to execute, and takes effort, because the testers limit the fault injection experiments and inject the minimum number of possible test cases. Fault injection enables testers to identify and address potential issues with a system under test before they become actual problems. In the automotive industry, failures can have serious hazards. In these systems, it is essential to ensure that the system can operate safely even in the presence of faults. We propose an approach using natural language processing (NLP) technologies to automatically derive the fault test cases from the functional safety requirements (FSRs) and execute them automatically by hardware-in-the-loop (HIL) in real time according to the black-box concept and the ISO 26262 standard. The approach demonstrates effectiveness in automatically identifying fault injection locations and conditions, simplifying the testing process, and providing a scalable solution for various safety-critical systems.

Software Failure Prediction Based On Program State and First-Error Characteristics

Abstract Evaluating program reliability against software faults involves the work of analyzing program failure against software faults as behavior on failures is fundamental for assessing reliability. Similar failure behavior reflects similar reliability assessment against software faults of same time–space distribution. Since program failure is determined by fault manifestations, working on similarities on fault manifestations helps figuring out failures in similarity as well. In this paper, we propose a novel method to characterize program behavior by defining fine-grained runtime states at assembly-level of code, aiming to capture the very first abnormal manifestation—first-error—after software fault being activated during program run. Failure prediction model and measurement is presented based on similarities reflected by the runtime behaviors obtained. Fault injection experiments are conducted to verify the prediction measurement by utilizing software faults of Orthogonal Defect Classification to inject and MiBench programs to perform. Over 15 000+ times of fault injection by considering type and location of fault were conducted into each program in order to quantify and analyze the sensitivity on first-error and similarity on failure accordingly. The results show that programs having particular structural features (e.g. massive operations with regards to calculations, nested function-calls, case structures and loop structures, etc.) are well characterized toward first-error behaviors by extracting fine-grained states. Similarities on first-error sensitivity can be represented better for these programs as well. Same trend is seen on failure behavior and its prediction measurement.

Intelligent fault diagnosis of ultrasonic motors based on graph-regularized CNN-BiLSTM

The ultrasonic motor (USM) is peculiarly prone to failure due to continuous high-frequency friction-related power transfer, whose failure mechanisms are remarkably different from traditional induction motors. Intelligent fault diagnosis provides a way to alarm and avoid catastrophic losses proactively. However, previous studies using deep learning usually ignore the inherent geometric structure of the signal distribution. This paper proposes an intelligent multi-signal fault diagnosis framework for USMs to restore the linear or nonlinear manifold structure by preserving the internal structure by integrating graph regularization with deep neural networks. Firstly, the one-dimensional CNN to learn spatial correlations and BiLSTM to exploit temporal dependencies are coalesced to build the deep neural network. Then, an improved k-nearest neighbor graph is proposed to protect the geometric structure information and force the latent features to be more concentrated within their classes. Moreover, the layer in the deep architecture to integrate graph regularization is designed to reduce computation cost, and an adaptive decay strategy is considered to adjust the coefficient of graph regularized automatically. A two-stage training algorithm is developed by considering the time to calculate the graph regularization term. Finally, the proposed multi-signal fault diagnosis framework is validated using datasets from the fault injection experiment of similar USMs in China’s Yutu rover of Chang’e lunar probe. Experimental results show that the proposed method can effectively discriminate different fault types.

Double laser-faults based PFA on cryptographic circuits with algebraic analysis

Cryptographic algorithms have been employed in a variety of fields as the primary method to protect information security. The security of a cryptographic algorithm is closely related to its operating environment and physical devices. Algebraic Persistent Fault Analysis (APFA) is a new fault analysis method for block ciphers proposed in CHES 2022, which utilizes the fault that persists in encryptions and introduces algebraic analysis in the fault analysis step. In the fault injection step, as the transistors of the integrated circuit are getting smaller and tighter, even high-precision devices may cause more than one fault per injection. However, more faults may lead to a more efficient attack in the fault analysis step. In this paper, APFA for double faults is proposed, which can deal with the double faults model and reduce the number of required ciphertexts. The practicality of our fault injection is validated by laser fault injection experiments on the SRAM embedded in an ATmega163L microcontroller. The effectiveness of our fault analysis is proven by successfully recovering the key of PRESENT-128 and AES-128. The number of ciphertexts needed for key recovery is reduced by 46% compared to PFA with a single fault.

Implementation of Highly Reliable Convolutional Neural Network with Low Overhead on Field-Programmable Gate Array

Due to the advantages of parallel architecture and low power consumption, a field-programmable gate array (FPGA) is typically utilized as the hardware for convolutional neural network (CNN) accelerators. However, SRAM-based FPGA devices are extremely susceptible to single-event upsets (SEUs) induced by space radiation. In this paper, a fault tolerance analysis and fault injection experiments are applied to a CNN accelerator, and the overall results show that SEUs occurring in a control unit (CTRL) lead to the highest system error rate, which is over 70%. After that, a hybrid hardening strategy consisting of a finite state machine error-correcting circuit (FSM-ECC) and a triple modular redundancy automatic hardening technique (TMR-AHT) is proposed in this paper to achieve a tradeoff between radiation reliability and design overhead. Moreover, the proposed methodology has very small workload and good migration ability. Finally, by full exploiting the fault tolerance property of CNNs, a highly reliable CNN accelerator with the proposed hybrid hardening strategy is implemented with Xilinx Zynq-7035. When BER is 2 × 10−6, the proposed hybrid hardening strategy reduces the whole system error rate by 78.95% with the overhead of an extra 20.7% of look-up tables (LUTs) and 20.9% of flip-flops (FFs).

Highly Efficient Self-checking Matrix Multiplication on Tiled AMX Accelerators

General Matrix Multiplication (GEMM) is a computationally expensive operation that is used in many applications such as machine learning. Hardware accelerators are increasingly popular for speeding up GEMM computation, with Tiled Matrix Multiplication (TMUL) in recent Intel processors being an example. Unfortunately, the TMUL hardware is susceptible to errors, necessitating online error detection. The Algorithm-based Error Detection (ABED) technique is a powerful technique to detect errors in matrix multiplications. In this article, we consider implementation of an ABED technique that integrates seamlessly with the TMUL hardware to minimize performance overhead. Unfortunately, rounding errors introduced by floating-point operations do not allow a straightforward implementation of ABED in TMUL. Previously an error bound was considered for addressing rounding errors in ABED. If the error detection threshold is set too low, it will a trigger false alarm, while a loose bound will allow errors to escape detection. In this article, we propose an adaptive error threshold that takes into account the TMUL input values to address the problem of false triggers and error escapes and provide a taxonomy of various error classes. This threshold is obtained from theoretical error analysis but is not easy to implement in hardware. Consequently, we relax the threshold such that it can be easily computed in hardware. While ABED ensures error-free computation, it does not guarantee full coverage of all hardware faults. To address this problem, we propose an algorithmic pattern generation technique to ensure full coverage for all hardware faults. To evaluate the benefits of our proposed solution, we conducted fault injection experiments and show that our approach does not produce any false alarms or detection escapes for observable errors. We conducted additional fault injection experiments on a Deep Neural Network (DNN) model and find that if a fault is not detected, it does not cause any misclassification.

An adaptive reweighted-Kurtogram for bearing fault diagnosis under strong external impulse noise

In vibration-based fault diagnosis of rotating machinery, strong impulse noise constitutes a widely encountered source of external interference, thereby posing significant challenges to the effective extraction of diagnostic information. In this context, an adaptive reweighted-Kurtogram (ARKurtogram) method is proposed. The bearing transient fault signatures are accurately extracted through two aspects – that is, precise frequency band division and robust band selection indicator. Specifically, we use the dual-tree complex wavelet packet transform (DTCWPT) as the band division tool and propose the sub-bands rearranged and ensemble DTCWPT which addresses the band disorder problem, yields far richer band division patterns and maintains excellent performance of DTCWPT in signal decomposition. Meanwhile, we introduce a robust indicator characterizing periodic fault impulses to accurately select the fault band. In comparison with the classical kurtosis and the advanced reweighted-kurtosis, this indicator is insensitive to strong external impulse noise and its calculation process contains a sliding window averaging approach which makes the calculation entirely automated without human intervention. The validity of the ARKurtogram is investigated with the analysis of vibration signals measured from fault-injection experiments. The comparisons with the classical fast Kurtogram method and the advanced Autogram method highlight the advantages of the method in extracting bearing transient fault signatures from complex signals with strong external impulse noise. It shows high potential in the application of industrial bearing fault diagnosis.

Investigating the impact of transient hardware faults on deep learning neural network inference

SummarySafety‐critical applications, such as autonomous vehicles, healthcare, and space applications, have witnessed widespread deployment of deep neural networks (DNNs). Inherent algorithmic inaccuracies have consistently been a prevalent cause of misclassifications, even in modern DNNs. Simultaneously, with an ongoing effort to minimize the footprint of contemporary chip design, there is a continual rise in the likelihood of transient hardware faults in deployed DNN models. Consequently, researchers have wondered the extent to which these faults contribute to DNN misclassifications compared to algorithmic inaccuracies. This article delves into the impact of DNN misclassifications caused by transient hardware faults and intrinsic algorithmic inaccuracies in safety‐critical applications. Initially, we enhance a cutting‐edge fault injector, TensorFI, for TensorFlow applications to facilitate fault injections on modern DNN non‐sequential models in a scalable manner. Subsequently, we analyse the DNN‐inferred outcomes based on our defined safety‐critical metrics. Finally, we conduct extensive fault injection experiments and a comprehensive analysis to achieve the following objectives: (1) investigate the impact of different target class groupings on DNN failures and (2) pinpoint the most vulnerable bit locations within tensors, as well as DNN layers accountable for the majority of safety‐critical misclassifications. Our findings regarding different grouping formations reveal that failures induced by transient hardware faults can have a substantially greater impact (with a probability up to 4 higher) on safety‐critical applications compared to those resulting from algorithmic inaccuracies. Additionally, our investigation demonstrates that higher order bit positions in tensors, as well as initial and final layers of DNNs, necessitate prioritized protection compared to other regions.

Generic Soft Error Data and Control Flow Error Detection by Instruction Duplication

Transient faults or soft errors are considered one of the most daunting reliability challenges for microprocessors. Software solutions for soft error protection are attractive because they can provide flexible and effective error protection. For instance, nZDC [1] state-of-the-art instruction duplication error protection scheme achieves a high degree of error detection by verifying the results of memory write operations and utilizes an effective control-flow checking mechanism. However, nZDC control-flow checking mechanism is architecture-dependent and suffers from some vulnerability holes. In this work, we address these issues by substituting nZDC control-flow checking mechanism with a general (ISA-independent) scheme and propose two transformations, coarse-grained scheduling, and asymmetric control-flow signatures, for hard-to-detect control flow errors. Fault injection experiments on different hardware components of synthesizable Verilog description of an OpenRISC-based microprocessor reveal that the proposed transformation shows 85% less silent data corruptions compared to nZDC. In addition, programs protected by the proposed scheme run on average around 37% faster than nZDC-protected programs.

Characterizing and Improving Resilience of Accelerators to Memory Errors in Autonomous Robots

Motion planning is a computationally intensive and well-studied problem in autonomous robots. However, motion planning hardware accelerators (MPA) must be soft-error resilient for deployment in safety-critical applications, and blanket application of traditional mitigation techniques is ill-suited due to cost, power, and performance overheads. We propose Collision Exposure Factor (CEF), a novel metric to assess the failure vulnerability of circuits processing spatial relationships, including motion planning. CEF is based on the insight that the safety violation probability increases with the surface area of the physical space exposed by a bit-flip. We evaluate CEF on four MPAs. We demonstrate empirically that CEF is correlated with safety violation probability, and that CEF-aware selective error mitigation provides 12.3 ×, 9.6 ×, and 4.2 × lower dangerous Failures-In-Time rate on average for the same amount of protected memory compared to uniform, bit-position, and access-frequency-aware selection of critical data. Furthermore, we show how to employ CEF to enable fault characterization using 23, 000 × fewer fault injection (FI) experiments than exhaustive FI, and evaluate our FI approach on different robots and MPAs. We demonstrate that CEF-aware FI can provide insights on vulnerable bits in an MPA while taking the same amount of time as uniform statistical FI. Finally, we use the CEF to formulate guidelines for designing soft-error resilient MPAs.

Efficient protection of polar decoders against Single Event Upsets (SEUs) on user memories

Polar codes are used in 5G systems for the transmission of control channels due to their excellent error correction capability for short sequences. CRC aided successive cancellation list (CA-SCL) decoders are commonly used in practical systems. When used in critical environments, e.g. space platforms, memories used in the polar decoder will suffer single-event upsets (SEUs) that can cause failures and disrupt communications. Therefore, the protection of polar decoders against SEUs on memories with low overhead is an important problem, especially for resource-limited on-board space systems. In this paper, efficient protection schemes are first proposed for each of the main memory components of CA-SAL decoders based on their algorithmic properties. Then FPGA implementation and hardware-based fault injection experiments are performed to evaluate the protection overhead and reliability of the proposed schemes. Experimental results show that, relative to the unprotected decoder, the reliability of the decoder is improved by more than two orders of magnitude, and the overhead of memory and computation are only about 1.26 and 1.23 times of those of unprotected decoder.

A Digital-Twin-Based Detection and Protection Framework for SDC-Induced Sinkhole and Grayhole Nodes in Satellite Networks

In the space environment, cosmic rays and high-energy particles may cause a single-event upset (SEU) during program execution, and further cause silent data corruption (SDC) errors in program outputs. After extensive research on SEU and SDC errors, it has been found that SDC errors in the routing program in satellite networks may lead to the emergence of Sinkhole (SH) and Grayhole (GH) nodes in the network, which may cause damage to satellite networks. To find and solve the problems in time, a digital-twin-based detection and protection framework for SDC-induced SH and GH nodes in satellite networks is proposed. First, the satellite network fault model under SEU and the generation mechanism of SH and GH nodes induced by SDC errors are described. Then, the data structure based on digital twins required by the proposed detection and protection framework is designed, and the detection methods of SH and GH nodes induced by SDC errors are proposed. SKT and LLFI simulation tools are used to build a simulated Iridium satellite network and carry out fault injection experiments. Experiment results show that the accuracy of the proposed detection method is 98–100%, and the additional time cost of routing convergence caused by the proposed framework is 3.1–28.2%. Compared with existing SH and GH detection methods, the proposed methods can timely and accurately detect faults during the routing update stage.

Efficient Protection of FPGA Implemented LDPC Decoders Against Single Event Upsets (SEUs) on Configuration Memories

Low Density Parity Check (LDPC) codes are used in 5G systems for traffic channels due to their excellent error correction capability for long sequences, and the Min-Sum algorithm is widely applied in practical implementations of LDPC decoders due to its low complexity. If the decoder is implemented on a SRAM-based field-programmable gate array (SRAM-FPGA), the radiation-induced single-event upsets (SEUs) can affect the operation of the LDPC decoder by corrupting the configuration memory, which can change the circuit functionality and will not be corrected unless the FPGA is reconfigured. Therefore, protection of LDPC decoders with low overhead is an important problem, especially for resource-limited on-board space systems. In this paper, an efficient Duplicate With Comparison (DWC) protection scheme is proposed based on the different distribution of the parity check sum of the LDPC decoder in the error-free case and the faulty case. In particular, the check sum accumulation number and threshold are optimized to achieve high detection probability with short delay. FPGA based implementation and hardware fault injection experiments are conducted to evaluate the performance of the proposed schemes. Experimental results show that, the effect of SEUs on the LDPC decoder can be completely eliminated by the proposed scheme with 2 times computational overhead and 1.69 times power consumption overhead compared to the unprotected decoder.

Mixed precision support in HPC applications: What about reliability?

In their quest for exascale and beyond, High-Performance Computing (HPC) systems continue becoming ever larger and more complex. Application developers, on the other hand, leverage novel methods to improve the efficiency of their own codes: a recent trend is the use of floating-point mixed precision, or the careful interlocking of single- and double-precision arithmetic, as a tool to improve performance as well as reduce network and memory boundedness. However, while it is known that modern HPC systems suffer hardware faults at daily rates, the impact of reduced precision on application reliability is yet to be explored. In this work we aim to fill this gap: first, we propose a qualitative survey to identify the branches of HPC where mixed precision is most popular. Second, we show the results of instruction-level fault injection experiments on a variety of representative HPC workloads, comparing vulnerability to Silent Data Errors (SDEs) under different numerical configurations. Our experiments indicate that use of single and mixed precision leads to comparatively more frequent and more severe SDEs, with concerning implications regarding their use on extreme-scale, fault-prone HPC platforms.

A Methodology for the Design of Fault Tolerant Parallel Digital Channelizers on SRAM-FPGAs

Digital channelizers (DCs) based on the Discrete Fourier Transform (DFT) and polyphase filter banks are widely used in on-board processing (OBP) platforms to extract narrowband sub-channels from a wideband signal efficiently. In high-capacity communication satellite platforms there are always multiple DCs extracting narrowband signals from multiple wideband signals in parallel. Field-programmable gate arrays (FPGAs) are a popular option for the implementation of DCs due to their parallel computing capabilities and good re-configurability, but FPGAs suffer single-event upsets (SEUs) on the space platform. This paper focuses on the efficient protection of parallel DCs with enhanced coding techniques. We first prove that a linear relationship between parallel DCs can be introduced and maintained among the multiple outputs. However, traditional coding schemes cannot be directly applied for the detection of faulty DCs due to the quantization noise introduced by fixed point implementations. To address this issue, we propose an enhanced coding scheme by averaging in the space and time domains to minimize the effect of quantization noise, introducing thresholds and a majority voter to further improve the detection probability. Both theoretical analysis and fault injection experiments prove the effectiveness of the proposed protection scheme. Experimental results show that all the SEUs that cause an SNR lower than 20dB can be detected and recovered, and the resource overheads are about 1.6 times and 1.3 times of that of the unprotected DCs for systems with 8 DCs and 16 DCs, respectively.

Capturing Request Execution Path for Understanding Service Behavior and Detecting Anomalies Without Code Instrumentation

With the increasing scale and complexity of cloud platforms and big-data analytics platforms, it is becoming more and more challenging to understand and diagnose the processing of a service request across multi-layer software stacks of such platforms. One way that helps to deal with this problem is to accurately capture the complete end-to-end execution path of service requests among all involved components. This paper presents REPTrace, a generic methodology for capturing such execution paths in a transparent fashion. Moreover, this paper demonstrates the effectiveness of REPTrace by presenting how REPTrace can be leveraged for knowledge extraction and anomaly detection on the platforms’ request processing. Our experimental results show that, REPTrace enables capturing a holistic view of the request processing across multiple layers of the platforms (which is missing in official documentation) and discovering important undocumented features of the platforms. Fault injection experiments show execution anomalies are detected with 93% precision and 96% recall with aid of REPTrace.

Systematic Reliability Evaluation of FPGA Implemented CNN Accelerators

Convolutional neural networks (CNN) have become essential for many scientific and industrial applications, such as image classification and pattern detection. Among the devices that can implement neural networks, SRAM based FPGAs are a popular option due to their excellent parallel computing capability and good flexibility. However, SRAM-FPGAs are susceptible to radiation effects, which limits its application on safety critical applications. In this paper, the reliability of an accelerator based on the advanced Instruction-Set Architecture is evaluated based on hardware fault injection experiments. Each main module of the accelerator is evaluated separately, and the impact of parallelism and model features on the accelerator reliability is also examined. The experimental results reveal some important conclusions in terms of general hardware reliability and also of the particular model reliability. First, over 99% of SEUs on the computation modules will cause accuracy loss, and the reliability improves for higher parallelism. Second, a large portion of SEUs on the data mover and the instruction scheduler will cause system corruptions due to abnormal interactions with the ARM or other modules. Third, nonlinear activation and pooling layers are effective in reducing the effect of SEUs on computation modules, so models that use these layers tend to be more robust. The results provide a deep understanding of the impact of errors on CNNs implemented on ISA based FPGA accelerators (e.g., the Xilinx DPU).

Run-time failure detection via non-intrusive event analysis in a large-scale cloud computing platform

Cloud computing systems fail in complex and unforeseen ways due to unexpected combinations of events and interactions among hardware and software components. These failures are especially problematic when they are silent, i.e., not accompanied by any explicit failure notification, hindering the timely detection and recovery.In this work, we propose an approach to run-time failure detection tailored for monitoring multi-tenant and concurrent cloud computing systems. The approach uses a non-intrusive form of event tracing, without manual changes to the system’s internals to propagate session identifiers (IDs), and builds a set of lightweight monitoring rules from fault-free executions. We evaluated the effectiveness of the approach in detecting failures in the context of the OpenStack cloud computing platform, a complex and “off-the-shelf” distributed system, by executing a campaign of fault injection experiments in a multi-tenant scenario. Our experiments show that the approach detects the failure with an F1 score (0.85) and accuracy (0.77) higher than the ones provided by the OpenStack failure logging mechanisms (0.53 and 0.50) and two non-session-aware run-time verification approaches (both lower than 0.15). Moreover, the approach significantly decreases the average time to detect failures at run-time (∼114 seconds) compared to the OpenStack logging mechanisms.

A fully blind and adaptive filter method to solve the inverse problem in vibration-based gear damage detection

In vibration-based gear damage detection, the fault signatures are usually corrupted by noise and convolution of vibration transmission channels. Deconvolution techniques have been widely studied to solve the inverse problem for restoring fault signatures. The mainstream methods, however, often fail due to the requirement of prior knowledge about fault signatures or the experience-based determination of filter parameters. In this paper, a fully blind and adaptive filter method is proposed. Specifically, a new criterion is defined, and the filter method is formulated as a maximization problem based on the criterion. The detailed deduction and pseudo-code for iteratively solving the problem are presented. Moreover, the filter method embeds an advanced swarm intelligence algorithm—that is, the grey wolf optimizer to achieve the adaptive determination of the optimal filter parameters according to the signals. In this context, it is a fully blind and adaptive filter method without requiring any prior knowledge and experiences about the target signatures to pre-determine filter parameters. Analysis results of a simulated signal and the comparisons with several state-of-the-art methods show the effectiveness of the proposed method for restoring the repetitive fault impulses from the mixed signal containing complex interferences. Fault-injection experiment and engineering application are carried out and the results demonstrate the performance of the proposed method for solving the inverse problem in gear damage detection.

A Spectral Boundary‐Integral Method for Faults and Fractures in a Poroelastic Solid: Simulations of a Rate‐and‐State Fault With Dilatancy, Compaction, and Fluid Injection

AbstractFluid‐fault interactions result in many two‐way coupled processes across a range of length scales, from the micron scale of the shear zone to the kilometer scale of the slip patch. The scale separation and complex coupling render fluid‐fault interactions challenging to simulate, yet they are key for our understanding of experimental data and induced seismicity. Here we present spectral boundary‐integral solutions for in‐plane interface sliding and opening in a poroelastic solid. We solve for fault slip in the presence of rate‐and‐state frictional properties, inelastic dilatancy, injection, and the coupling of a shear zone and a diffusive poroelastic bulk. The shear localization zone is treated as having a finite width and non‐constant pore pressure, albeit with a simplified mathematical representation. The dimension of the 2D plane strain problem is reduced to a 1D problem resulting in increased computational efficiency and incorporation of small‐scale shear‐zone physics into the boundary conditions. We apply the method to data from a fault injection experiment that has been previously studied with modeling. We explore the influence of bulk poroelastic response, bulk diffusivity in addition to inelastic dilatancy on fault slip during injection. Dilatancy not only alters drastically the stability of fault slip but also the nature of pore pressure evolution on the fault, causing significant deviation from the standard square‐root‐of‐time diffusion. More surprisingly, varying the bulk's poroelastic response (by using different values of the undrained Poisson's ratio) and bulk hydraulic diffusivity can be as critical in determining rupture stability as the inelastic dilatancy.