Silent Data Corruption Research Articles

Analysing the radiation reliability, performance and energy consumption of low-power SoC through heterogeneous parallelism

This study focuses on the low-power Tegra X1 System-on-Chip (SoC) from the Jetson Nano Developer Kit, which is increasingly used in various environments and tasks. As these SoCs grow in prevalence, it becomes crucial to analyse their computational performance, energy consumption, and reliability, especially for safety-critical applications. A key factor examined in this paper is the SoC’s neutron radiation tolerance. This is explored by subjecting a parallel version of matrix multiplication, which has been offloaded to various hardware components via OpenMP, to neutron irradiation. Through this approach, this researcher establishes a correlation between the SoC’s reliability and its computational and energy performance. The analysis enables the identification of an optimal workload distribution strategy, considering factors such as execution time, energy efficiency, and system reliability. Experimental results reveal that, while the GPU executes matrix multiplication tasks more rapidly and efficiently than the CPU, using both components only marginally reduces execution time. Interestingly, GPU usage significantly increases the SoC’s critical section, leading to an escalated error rate for both Detected Unrecoverable Errors (DUE) and Silent Data Corruptions (SDC), with the CPU showing a higher average number of affected elements per SDC.

Understanding Silent Data Corruption in Processors for Mitigating its Effects

Silent Data Corruption (SDC) in processors can lead to various application-level issues, such as incorrect calculations and even data loss. Since traditional techniques are not effective in detecting these errors, it is very hard to address problems caused by SDCs in processors. For the same reason, knowledge about these SDCs in the wild is limited. In this paper, we conduct an extensive study on CPU SDCs in a large production CPU population, encompassing over one million processors. In addition to collecting overall statistics, we perform a detailed study to understand 1) whether certain processor features are particularly vulnerable and their potential impacts on applications; 2) the reproducibility of CPU SDCs and the triggering conditions (e.g., temperature) of those less reproducible SDCs; and 3) the challenges to mitigate and handle CPU SDCs. We further investigate the implications which our observations obtained from the above researches have, on the SDC fault models, SDC mitigation strategies and the future research fields. In addition, we design an efficient SDC mitigation approach called Farron, which uses prioritized testing to detect highly reproducible SDCs and temperature control to mitigate less reproducible SDCs. Our experimental results indicate that Farron can achieve better coverage of CPU SDCs with lower overall overhead, compared to the baseline used in Alibaba Cloud. This demonstrates that our observations are able to assist in SDC mitigation.

ParDiff: Practical Static Differential Analysis of Network Protocol Parsers

Countless devices all over the world are connected by networks and communicated via network protocols. Just like common software, protocol implementations suffer from bugs, many of which only cause silent data corruption instead of crashes. Hence, existing automated bug-finding techniques focused on memory safety, such as fuzzing, can hardly detect them. In this work, we propose a static differential analysis called ParDiff to find protocol implementation bugs, especially silent ones hidden in message parsers. Our key observation is that a network protocol often has multiple implementations and any semantic discrepancy between them may indicate bugs. However, different implementations are often written in disparate styles, e.g., using different data structures or written with different control structures, making it challenging to directly compare two implementations of even the same protocol. To exploit this observation and effectively compare multiple protocol implementations, ParDiff (1) automatically extracts finite state machines from programs to represent protocol format specifications, and (2) then leverages bisimulation and SMT solvers to find fine-grained and semantic inconsistencies between them. We have extensively evaluated ParDiff using 14 network protocols. The results show that ParDiff outperforms both differential symbolic execution and differential fuzzing tools. To date, we have detected 41 bugs with 25 confirmed by developers.

Can GPU performance increase faster than the code error rate?

Graphics processing units (GPUs) are the reference architecture to accelerate high-performance computing applications and the training/interference of convolutional neural networks. For both these domains, performance and reliability are two of the main constraints. It is believed that the only way to increase reliability is to sacrifice performance, e.g., using redundancies. We show in this paper that this is not always the case. As a very promising result, we found that most GPUs performance improvements also bring the benefit of increasing the number of executions correctly completed before experiencing a silent data corruption (SDC). We consider four different common GPUs’ performance optimizations: architectural solutions, software implementations, compiler optimizations, and threads degree of parallelism. We compare different implementations of a variety of parallel codes and, through beam experiments and applications profiling, we show that the performance improvement typically (but not necessarily) increases the GPU SDC rate. Nevertheless, for the vast majority of the configurations the performance gain is much higher than the SDC rate increase, allowing to process a higher amount of correct data. As we show, the programmer choices can increase up to 25×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$25\ imes {}$$\\end{document} the number of correctly completed executions without redesigning the algorithm nor including specific hardening solutions.

BiGResi: Robust bit-level fault injection framework for assessing intrinsic software resilience against soft errors

Radiation-induced soft errors, despite rare, pose a significant threat to the reliability of systems. Assessing the intrinsic resilience of software to soft errors is therefore essential for building fault-tolerant systems cost-effectively. Analytical models, while fast, can be imprecise. In contrast, Fault Injection (FI) has been successfully applied as a mature method for reliability assessment. While high-level FI offers less accuracy, existing low-level techniques can enhance resilience assessment accuracy by sacrificing some desirable features like fault coverage and intrusiveness. Furthermore, these techniques are often driven by random FI campaigns, making establishing a clear correlation between application characteristics and resilience challenging. This paper presents BiGResi, a versatile software-based framework for assessing software resilience. BiGResi overcomes the limitations of random, instruction type-agnostic FI techniques by evaluating resilience at a low-level granularity, considering instruction type and bit location. Furthermore, it targets the instruction set architecture (ISA), enhancing assessment accuracy by revealing architecturally visible faults. BiGResi employs a timing-based FI mechanism with negligible modifications to the target software, minimizing intrusiveness and ensuring near-native speed. BiGResi’s accuracy is empirically evaluated through many FI campaigns targeting different benchmarks with diverse characteristics. We observed that instruction types, ISA encoding bits, and bit location are key factors to consider when assessing software resilience. Finally, BiGResi’s effectiveness is demonstrated by selectively applying instruction protection, resulting in an average reduction of silent data corruptions (SDCs) by 73.80%, with a performance overhead of 15.46%. Furthermore, allowing a slightly higher overhead of 22% can improve the SDC detection rate by up to 93.83%.

Reduced On-chip Storage of Seeds for Built-in Test Generation

Logic built-in self-test ( LBIST ) approaches use an on-chip logic block for test generation and thus enable in-field testing. Recent reports of silent data corruption underline the importance of in-field testing. In a class of storage-based LBIST approaches, compressed tests are stored on-chip and decompressed by an on-chip decompression logic. The on-chip storage requirements may become a bottleneck when the number of compressed tests is large. In this case, using each compressed test for applying several different tests allows the storage requirements to be reduced. However, producing different tests from each compressed test has a hardware overhead. This article suggests a new on-chip storage scheme for compressed tests that eliminates the additional hardware overhead. Under the new storage scheme, a set of N B -bit compressed tests targeting a set of faults F 0 is translated into a sequence S of N ⋅ B bits. Every B consecutive bits of S are considered as a compressed test. The sequence S thus yields close to N ⋅ B compressed tests, magnifying the test data stored in S almost B times. Taking advantage of the extra tests, the article describes a software procedure that is applied offline to reduce S without losing fault coverage of F 0 . Experimental results for benchmark circuits demonstrate significant reductions in the storage requirements of S and significant increases in the fault coverage of a second set of faults, F 1 .

A Visual Comparison of Silent Error Propagation.

High-performance computing (HPC) systems play a critical role in facilitating scientific discoveries. Their scale and complexity (e.g., the number of computational units and software stack) continue to grow as new systems are expected to process increasingly more data and reduce computing time. However, with more processing elements, the probability that these systems will experience a random bit-flip error that corrupts a program's output also increases, which is often recognized as silent data corruption. Analyzing the resiliency of HPC applications in extreme-scale computing to silent data corruption is crucial but difficult. An HPC application often contains a large number of computation units that need to be tested, and error propagation caused by error corruption is complex and difficult to interpret. To accommodate this challenge, we propose an interactive visualization system that helps HPC researchers understand the resiliency of HPC applications and compare their error propagation. Our system models an application's error propagation to study a program's resiliency by constructing and visualizing its fault tolerance boundary. Coordinating with multiple interactive designs, our system enables domain experts to efficiently explore the complicated spatial and temporal correlation between error propagations. At the end, the system integrated a nonmonotonic error propagation analysis with an adjustable graph propagation visualization to help domain experts examine the details of error propagation and answer such questions as why an error is mitigated or amplified by program execution.

Generic Soft Error Data and Control Flow Error Detection by Instruction Duplication

Transient faults or soft errors are considered one of the most daunting reliability challenges for microprocessors. Software solutions for soft error protection are attractive because they can provide flexible and effective error protection. For instance, nZDC [1] state-of-the-art instruction duplication error protection scheme achieves a high degree of error detection by verifying the results of memory write operations and utilizes an effective control-flow checking mechanism. However, nZDC control-flow checking mechanism is architecture-dependent and suffers from some vulnerability holes. In this work, we address these issues by substituting nZDC control-flow checking mechanism with a general (ISA-independent) scheme and propose two transformations, coarse-grained scheduling, and asymmetric control-flow signatures, for hard-to-detect control flow errors. Fault injection experiments on different hardware components of synthesizable Verilog description of an OpenRISC-based microprocessor reveal that the proposed transformation shows 85% less silent data corruptions compared to nZDC. In addition, programs protected by the proposed scheme run on average around 37% faster than nZDC-protected programs.

Silent Data Corruptions: Microarchitectural Perspectives

Today more than ever before, academia, manufacturers, and hyperscalers acknowledge the major challenge of silent data corruptions (SDCs) and aim on solutions to minimize its impact by avoiding, detecting, and mitigating SDCs. Recent studies on large scale datacenters conducted by Meta and Google report an unexpected rate of silent data corruption incidents that are attributed to modern microprocessor generations. Despite the acknowledged severity of the phenomenon, particularly at the datacenter scale, there is no in-depth analysis of the microarchitectural locations in a complex microprocessor that are more likely to generate an SDC at the program outputs. In this paper, we present a detailed analysis of the faulty behavior of many critical microarchitectural structures of a modern out-of-order microprocessor generating silent data corruptions. Our analysis unveils several observations, including: (i) the magnitude of silent data corruptions attributed to different hardware structures, (ii) the instruction-related parameters that are more likely to result in a silent data corruption, (iii) the extent to which the operating system affects the silent data corruption occurrences, and (iv) the byte positions of a word which are more likely to result in silent data corruptions. Collectively, such findings can assist decisions for hardware and software schemes for the reduction of the likelihood of silent data corruptions generation.

A Digital-Twin-Based Detection and Protection Framework for SDC-Induced Sinkhole and Grayhole Nodes in Satellite Networks

In the space environment, cosmic rays and high-energy particles may cause a single-event upset (SEU) during program execution, and further cause silent data corruption (SDC) errors in program outputs. After extensive research on SEU and SDC errors, it has been found that SDC errors in the routing program in satellite networks may lead to the emergence of Sinkhole (SH) and Grayhole (GH) nodes in the network, which may cause damage to satellite networks. To find and solve the problems in time, a digital-twin-based detection and protection framework for SDC-induced SH and GH nodes in satellite networks is proposed. First, the satellite network fault model under SEU and the generation mechanism of SH and GH nodes induced by SDC errors are described. Then, the data structure based on digital twins required by the proposed detection and protection framework is designed, and the detection methods of SH and GH nodes induced by SDC errors are proposed. SKT and LLFI simulation tools are used to build a simulated Iridium satellite network and carry out fault injection experiments. Experiment results show that the accuracy of the proposed detection method is 98–100%, and the additional time cost of routing convergence caused by the proposed framework is 3.1–28.2%. Compared with existing SH and GH detection methods, the proposed methods can timely and accurately detect faults during the routing update stage.

Bare-Metal Redundant Multi-Threading on Multicore SoCs Under Neutron Irradiation

A software technique is presented to protect commercial multi-core microprocessors against radiation-induced soft errors. Important time overheads associated with conventional software redundancy techniques limit the feasibility of advanced critical electronic systems. In our approach, redundant bare-metal threads are used, so that critical computation is distributed over the different micro-processor cores. In doing so, software redundancy can be applied to Commercial Off-The-Shelf (COTS) micro-processors without incurring high-performance penalties. The proposed technique was evaluated using a low-cost single board computer (Raspberry Pi 4) under neutron irradiation. The results showed that the Redundant Multi-Threading versions detected and recovered all the Silent Data Corruption (SDC) events, and only increased HANG sensitivity with respect to the unhardened original versions. In addition, higher Mean Work to Failure (MWTF) estimations are achieved with our bare-metal technique than with the state-of-the-art bare-metal software-based techniques that only implement temporal redundancy.

Assessment of Radiation-Induced Soft Errors on Lightweight Cryptography Algorithms Running on a Resource-Constrained Device

Most safety-critical edge-computing devices rely on lightweight cryptography (LWC) algorithms to provide security at minimum power and performance overhead. LWC algorithms are traditionally embedded as a hardware component, but with the advance of the Internet of Things (IoT), emerging firmware is more likely to support cryptography algorithms to comply with different security levels and industry-standards. This is the first work to present the soft error assessment of five cryptography algorithms executing in a low-power microprocessor running under neutron radiation, considering electronic code book (ECB) and counter (CTR) mode of operation implementations. Results obtained from two neutron radiation tests suggest that: ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</i> ) the NOEKEON algorithm gives the best relative soft error reliability, performance, power efficiency and memory footprint utilisation trade-offs between the five algorithms considering both ECB and CTR implementations, and ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ii</i> ) cryptography solutions based on the counter mode of operation present better FIT rate for silent data corruption (SDC) and crash w.r.t. ECB implementations.

Multi-bit Data Flow Error Detection Method Based on SDC Vulnerability Analysis

One of the most difficult data flow errors to detect caused by single-event upsets in space radiation is the Silent Data Corruption (SDC). To solve the problem of multi-bit upsets causing program SDC, an instruction multi-bit SDC vulnerability prediction model based on one-class support vector machine classification is built using SDC vulnerability analysis, which has more accurate vulnerability instruction identification capabilities. By hardening the program with selective instruction redundancy, we propose a multi-bit data flow error detection method for detecting SDC error (SDCVA-OCSVM), aiming to protect the data in the memory or register used by the program. We have also verified the effectiveness of the method through comparative experiments. The method has been verified to have a higher error detection rate and lower code size and time overhead.

Automated Model Hardening with Reinforcement Learning for On-Orbit Object Detectors with Convolutional Neural Networks

On-orbit object detection has received extensive attention in the field of artificial intelligence (AI) in space research. Deep-learning-based object-detection algorithms are often computationally intensive and rely on high-performance devices to run. However, those devices usually lack space-qualified versions, and they can hardly meet the reliability requirement if directly deployed on a satellite platform, due to software errors induced by the space environment. In this paper, we evaluated the impact of space-environment-induced software errors on object-detection algorithms through large-scale fault injection tests. Aside from silent data corruption (SDC), we propose an extended criterial SDC-0.1 to better quantify the effect of the transient faults on the object-detection algorithms. Considering that a bit-flip error could cause severe detection result corruption in many cases, we propose a novel automated model hardening with reinforcement learning (AMHR) framework to solve this problem. AMHR searches for error-sensitive kernels in a convolutional neural network (CNN) through trial and error with a deep deterministic policy gradient (DDPG) agent and has fine-grained modular-level redundancy to increase the fault tolerance of the CNN-based object detectors. Compared to other selective hardening methods, AMHR achieved the lowest SDC-0.1 rates for various detectors and could tremendously improve the mean average precision (mAP) of the SSD detector by 28.8 in the presence of multiple errors.

Soft Error Effects on Arm Microprocessors: Early Estimations versus Chip Measurements

Extensive research efforts are being carried out to evaluate and improve the reliability of computing devices either through beam experiments or simulation-based fault injection. Unfortunately, it is still largely unclear to which extend fault injection can provide an accurate error rate estimation at early stages and if beam experiments can be used to identify the weakest resources in a device. The importance and challenges associated with a timely, but yet realistic reliability evaluation grow with the increase of complexity in both the hardware domain, with the integration of different types of cores in an SoC (System-on-Chip), and the software domain, with the OS (operating system) required to take full advantage of the available resources. In this paper, we combine and analyze data gathered with extensive beam experiments (on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">final</i> physical CPU hardware) and microarchitectural fault injections (on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">early</i> microarchitectural CPU models). We target a standalone Arm Cortex-A5 CPU and an Arm Cortex-A9 CPU integrated into an SoC and evaluate their reliability in bare-metal and Linux-based configurations. Combining experimental data that covers more than 18 million years of device time with the result of more than 176,000 injections we find that both the SoC integration and the presence of the OS increase the system DUEs (Detected Unrecoverable Errors) rate (for different reasons) but do not significantly impact the SDCs (Silent Data Corruptions) rate which is solely attributed to the CPU core. Our reliability analysis demonstrates that even considering SoC integration and OS inclusion, early, pre-silicon microarchitecture-level fault injection delivers accurate SDC rates estimations and lower bounds for the DUE rates.

CAFI: A Configurable location-Aware Fault Injection technique for software reliability assessment against soft errors

The rapid development of processor manufacturing technologies has encouraged most designers to highlight the processors' resilience to errors. Regarding embedded systems in harsh environments, temporary faults, such as single-event upsets (SEUs) or single-event multiple upsets (SEMUs), induced by radiation effects on a memory cell or a combinational logic circuit, can cause perturbations in the system's behavior, which in turn can cause catastrophic consequences. In this context, Fault Injection (FI) has been successfully applied as a mature method to assess reliability against faults and reveal the system's deficiencies to be protected cost-effectively. Although high-level software FI techniques are less accurate, the high accuracy of fault injection at a low level usually comes at the expense of desirable characteristics related to portability, flexibility, and intrusiveness. Hence, this paper proposes a Configurable location-Aware Fault Injection (CAFI) technique. CAFI is a software-based technique that facilitates emulating different kinds of transient hardware faults, e.g., SEUs and SEMUs, at the software level. Therefore, it supports flexibility in conducting reliability assessment studies with varying fault models. It operates at the binary code level and exploits a timing-based mechanism to inject faults at run-time in a negligible intrusive manner. CAFI can inject faults at different granularity levels to maximize fault activation. In detail, a fine-grained into a specific instruction field and a coarse-grained into the whole system's software. CAFI requires negligible modifications to the target software under test and allows it to run at near-native speed. The effectiveness of CAFI is evaluated by conducting many fault injection experiments applying to different real-world benchmark programs. Moreover, the accuracy of CAFI is quantified with respect to high-level software fault injection. For this aim, a practical prototype of CAFI is implemented on the x86 architecture employing an Intel core i7 processor with 16GB RAM. Based on a total of 108,000 fault injection experiments, the rate of program crashes caused by CAFI is slightly more than 18% higher than that caused by high-level software fault injection. On the other hand, there are no significant differences between the silent data corruptions (SDCs), results obtained by CAFI, and high-level software fault injections, making CAFI applicable for studying crash- and SDC-causing errors.

An efficient structure to improve the reliability of deep neural networks on ARMs

As Deep Neural Networks (DNNs) become more pervasive in safety-critical embedded systems, improving the soft error resilience of DNNs will grow increasingly important. This paper proposes a Distribution-based Error Detector (DED) to improve DNN's reliability. We compare the proposed approach with the regularization method and the typical Symptom-based Error Detector (SED). From the perspective of the bit error resilience, DED provides the highest fault coverage. Our results show that DED DNNs' Silent Data Corruption rates are less than 0.02 even if error bit rates are up to 1. Further, regarding Architecture Vulnerability Factor (AVF) results, we observe that the regularization method and the SED cannot improve the error resilience of register files for quantized DNNs. On the contrary, DED can reduce the SDC AVF by order of magnitude. In addition, DED can increase Mean Work To Failure (MWTF) by more than 19×, while the regularization method and the SED only increase MWTF by less than 2×.

Characterizing and Mitigating Soft Errors in GPU DRAM

While graphics processing units (GPUs) are used in high-reliability systems, wide GPU dynamic random-access memory (DRAM) interfaces make error protection difficult, as wide-device correction through error checking and correcting (ECC) is expensive and impractical. This challenge is compounded by worsening relative rates of multibit DRAM errors and increasing GPU memory capacities. This work uses high-energy neutron beam tests to inform the design and evaluation of GPU DRAM error-protection mechanisms. Based on observed locality in multibit error patterns, we propose several novel ECC schemes to decrease the silent data corruption (SDC) risk by up to five orders of magnitude relative to single-bit-error-correcting and double-bit-error-detecting (SEC-DED) ECC, while also reducing the number of uncorrectable errors by up to 7.87×. We compare novel binary and symbol-based ECC organizations that differ in their design complexity and hardware overheads, ultimately recommending two promising organizations. These schemes replace SEC-DED ECC with no additional redundancy, likely no performance degradation, and modest area and complexity costs.

Characterization of Program Behavior under Faulty Instruction Encoding

As process technology scales, electronic devices become more susceptible to soft errors. Soft errors can lead to silent data corruptions (SDCs), seriously compromising the reliability of a system. Researchers have explored error resilient encodings, which leverage crash patterns to detect SDCs. Despite its importance, much still remains to be determined regarding how errors propagate to cause SDCs or crashes. Understanding error propagation patterns could lead to more efficient implementation of error detection. An experimental study of program behavior in the presence of faulty instruction encoding under the IA-32 architecture is described in this study. Extensive fault injection experiments including over 70,000 faults were conducted, targeting all fields of instruction encoding. The analysis of the obtained data shows the following: (1) If the alignment of an instruction sequence is not preserved after injection, it causes crashes in a high probability (93.2%). (2) The SDC rate of an alignment-preserved category is close to that of a typical data injection. The SDC-prone fields include the opcode field, reg field, and immediate field. (3) Several crash patterns, such as violation of calling conventions, are revealed to extend the detection methods. These findings help us identify the vulnerable parts of instruction encoding, which need to be protected against soft errors. By applying the implications provided by the findings, we discuss feasible modifications, including swapping reg encodings, to reduce SDC rate, thus increasing the resilience of instruction set to soft errors.

The Impact of Terrestrial Radiation on FPGAs in Data Centers

Field programmable gate arrays (FPGAs) are used in large numbers in data centers around the world. They are used for cloud computing and computer networking. The most common type of FPGA used in data centers are re-programmable SRAM-based FPGAs. These devices offer potential performance and power consumption savings. A single device also carries a small susceptibility to radiation-induced soft errors, which can lead to unexpected behavior. This article examines the impact of terrestrial radiation on FPGAs in data centers. Results from artificial fault injection and accelerated radiation testing on several data-center-like FPGA applications are compared. A new fault injection scheme provides results that are more similar to radiation testing. Silent data corruption (SDC) is the most commonly observed failure mode followed by FPGA unavailable and host unresponsive. A hypothetical deployment of 100,000 FPGAs in Denver, Colorado, will experience upsets in configuration memory every half-hour on average and SDC failures every 0.5–11 days on average.