Silent Data Corruption Research Articles

G-SEAP: Analyzing and characterizing soft-error aware approximation in GPGPUs

As General-Purpose Graphics Processing Units (GPGPUs) become pervasive for the High-Performance Computing (HPC), ensuring that programs can be protected from soft errors has become increasingly important. Soft errors may cause Silent Data Corruptions (SDCs), which produces erroneous execution results silently. Due to the massive parallelism of GPGPUs, fully protecting them against soft errors introduces nontrivial overhead. Fortunately, imprecise execution outcomes are inherently tolerable for some HPC programs due to the nature of these applications. Leveraging the feature, selective soft error protection can be applied to reduce energy consumptions.In this work, we first propose a GPGPU-based Soft-Error aware APproximation analysis framework (G-SEAP) to characterize the approximation characteristics of soft errors. Based on G-SEAP, we perform an exhaustive analysis for 17 representative HPC benchmarks and observe 72.7% of SDCs on average are approximable. We also observe that the dataflow of application, kernel function reliability requirement, instruction-type, and data bit-location are all important factors for program’s correctness. Lastly, according to the observations, we further design an approximate Error Correction Codes (ECCs) mechanism and an approximate instruction duplication technique to illustrate how G-SEAP provides useful guidance for energy-efficient soft-error elimination in GPGPUs.

Improving the Accuracy of IR-Level Fault Injection

Fault injection (FI) is a commonly used experimental technique to evaluate the resilience of software techniques for tolerating hardware faults. Software-implemented FI can be performed at different levels of abstraction in the system stack; FI performed at the compiler&#x2019;s intermediate representation (IR) level has the advantage that it is closer to the program being evaluated and is hence easier to derive insights from for the design of software fault-tolerance mechanisms. Unfortunately, it is not clear how accurate IR-level FI is vis-a-vis FI performed at the assembly code level, and prior work has presented contradictory findings. In this article, we perform a comprehensive evaluation of the accuracy of IR-level FI across a range of benchmark programs and compiler optimization levels. Our results show that IR-level FI is as accurate as assembly-level FI for silent data corruption (SDC) probability estimation across different benchmarks and optimization levels. Further, we present a machine-learning-based technique for improving the accuracy of <i>crash</i> probability measurements made by IR-level FI, which takes advantage of an observed correlation between program crash probabilities and instructions that operate on memory address values. We find that the machine learning technique provides comparable accuracy for IR-level FI as assembly code level FI for program crashes.

Analyzing the Effects of Bugs on Software Interfaces

Critical systems that integrate software components (e.g., from third-parties) need to address the risk of residual software defects in these components. Software fault injection is an experimental solution to gauge such risk. Many error models have been proposed for emulating faulty components, such as by injecting error codes and exceptions, or by corrupting data with bit-flips, boundary values, and random values. Even if these error models have been able to find breaches in fragile systems, it is unclear whether these errors are in fact representative of software faults. To pursue this open question, we propose a methodology to analyze how software faults in C/C++ software components turn into errors at components’ interfaces ( interface error propagation ), and present an experimental analysis on what, where, and when to inject interface errors. The results point out that the traditional error models, as used so far, do not accurately emulate software faults, but that richer interface errors need to be injected, by: injecting both fail-stop behaviors and data corruptions; targeting larger amounts of corrupted data structures; emulating silent data corruptions not signaled by the component; combining bit-flips, boundary values, and data perturbations.

Quantitative evaluation of fault propagation in a commercial cloud system

As semiconductor technology scales into the nano regime, hardware faults have been threats against computational devices. Cloud systems are incorporating more and more computing density and energy into themselves; thus, fundamental research on topics such as dependability validation is needed, in order to verify the robustness of clouds for sensor networks. However, dependability evaluation studies have often been carried out beyond isolated physical systems, such as processors, sensors, and single boards with or without operating system hosts. These studies have been performed using inaccurate simulations instead of validating complete cloud software stacks (firmware, hypervisor, operating system hosts and workloads) as a whole. In this article, we describe the implementation of a fault injection tool, which validates the dependability of a commercial cloud software stack. Hardware faults induced by high energy density environments can be injected; the fault propagation through the cloud software stack is traced, and quantitatively evaluated. Experimental results show that the integrated fault detection mechanism of the cloud system, such as fatal trap detectors, has left a detection margin of 20% silent data corruption to narrow down. We additionally propose two detection mechanisms, which proved good performance in fault detection of cloud systems.

The Effects of Soft Errors and Mitigation Strategies for Virtualization Servers

Virtualized servers compose the majority of cloud computing environments, where these nodes are used to host multiple clients over the same hardware. Many organizations run online applications by hiring elastic computing resources in order to match demand while reducing fixed costs. However, such organizations are unlikely to take advantage of these benefits for critical applications, as it would expose them to several risks. Among other threats, soft errors are a concern in large-scale reliable servers and are expected to become more frequent as a consequence of smaller transistors and lower operating voltages of integrated circuits. This article characterizes virtualized servers of cloud environments in presence of soft errors. Using fault injection, we collect experimental data to determine the failure modes of applications, operating systems, VMs, and hypervisor. The analysis exposes distinct failure modes, ranging from crash failures of a single virtual machine to silent data corruption in permanent storage. The most frequent failure mode, observed in 10–30 percent of injected errors, consists of a hang affecting multiple virtual machines. Given that such failures are a primary cause of downtime, we develop and evaluate a recovery mechanism which uses online testing and recovers a server from all hangs by rebooting its hypervisor.

Recovery algorithm to correct silent data corruption of synaptic storage in convolutional neural networks

With the surge of computational power and efficient energy consumption management on embedded devices, embedded processing has grown exponentially during the last decade. In particular, computer vision has become prevalent in real-time embedded systems, which have always been a victim of transient fault due to its pervasive presence in harsh environments. Convolutional Neural Networks (CNN) are popular in the domain of embedded vision (computer vision in embedded systems) given the success they have shown. One problem encountered is that a pre-trained CNN on embedded devices is vastly affected by Silent Data Corruption (SDC). SDC refers to undetected data corruption that causes errors in data without any indication that the data is incorrect, and thus goes undetected. In this paper, we propose a software-based approach to recover the corrupted bits of a pre-trained CNN due to SDC. Our approach uses a rule-mining algorithm and we conduct experiments on the propagation of error through the topology of the CNN in order to detect the association of the bits for the weights of the pre-trained CNN. This approach increases the robustness of safety-critical embedded vision applications in volatile conditions. A proof of concept has been conducted for a combination of a CNN and a vision data set. We have successfully established the effectiveness of this approach for a very high level of SDC. The proposed approach can further be extended to other networks and data sets.

Analyzing and Increasing the Reliability of Convolutional Neural Networks on GPUs

Graphics processing units (GPUs) are playing a critical role in convolutional neural networks (CNNs) for image detection. As GPU-enabled CNNs move into safety-critical environments, reliability is becoming a growing concern. In this paper, we evaluate and propose strategies to improve the reliability of object detection algorithms, as run on three NVIDIA GPU architectures. We consider three algorithms: 1) you only look once; 2) a faster region-based CNN (Faster R-CNN); and 3) a residual network, exposing live hardware to neutron beams. We complement our beam experiments with fault injection to better characterize fault propagation in CNNs. We show that a single fault occurring in a GPU tends to propagate to multiple active threads, significantly reducing the reliability of a CNN. Moreover, relying on error correcting codes dramatically reduces the number of silent data corruptions (SDCs), but does not reduce the number of critical errors (i.e., errors that could potentially impact safety-critical applications). Based on observations on how faults propagate on GPU architectures, we propose effective strategies to improve CNN reliability. We also consider the benefits of using an algorithm-based fault-tolerance technique for matrix multiplication, which can correct more than 87% of the critical SDCs in a CNN, while redesigning maxpool layers of the CNN to detect up to 98% of critical SDCs.

Assessing the Reliability of Successive Approximate Computing Algorithms under Fault Injection

This work presents two fault injection and dependability test methodologies exploring the fault tolerance of successive approximation algorithms. This type of approximate computing algorithm can present an inherent fault tolerance, converging to a final correct output even under faults affecting processed data. A set of algorithms was implemented as embedded software in the ARM Cortex A9 processor of Xilinx Zynq-7000 series board. Experiments consist of exposing the decapsulated processor to laser beams targeting the data cache memory and emulation fault injections at the register file. Results show that successive approximation is effective in protecting the output from faults injected at the data cache memory, but not from the ones injected at the register file. The experiments also show that most of the silent data corruption errors provoked by data cache fault injections are not significant and can be accepted as correct by merely tolerating a result variation of as little as 1%.

Identifying SDC-Causing Instructions Based on Random Forests Algorithm

Silent Data Corruptions (SDCs) is a serious reliability issue in many domains of computer system. The identification and protection of the program instructions that cause SDCs is one of the research hotspots in computer reliability field at present. A lot of solutions have already been proposed to solve this problem. However, many of them are hard to be applied widely due to time-consuming and expensive costs. This paper proposes an intelligent approach named SDCPredictor to identify the instructions that cause SDCs. SDCPredictor identifies SDC-causing Instructions depending on analyzing the static and dynamic features of instructions rather than fault injections. The experimental results demonstrate that SDCPredictor is highly accurate in predicting the SDCs proneness. It can achieve higher fault coverage than previous similar techniques in a moderate time cost.

Dependability Analysis of Data Storage Systems in Presence of Soft Errors

In recent years, high availability and reliability of Data Storage Systems (DSS) have been significantly threatened by soft errors occurring in storage controllers. Due to their specific functionality and hardware-software stack, error propagation and manifestation in DSS is quite different from general-purpose computing architectures. To our knowledge, no previous study has examined the system-level effects of soft errors on the availability and reliability of data storage systems. In this paper, we first analyze the effects of soft errors occurring in the server processors of storage controllers on the entire storage system dependability. To this end, we implemented the major functions of a typical data storage system controller, running on a full stack of storage system operating system, and developed a framework to perform fault injection experiments using a full system simulator. We then propose a new metric, Storage System Vulnerability Factor (SSVF), to accurately capture the impact of soft errors in storage systems. By conducting extensive experiments, it is revealed that depending on the controller configuration, up to 40% of cache memory contains end-user data where any unrecoverable soft errors in this part will result in Data Loss (DL) in an irreversible manner. However, soft errors in the rest of cache memory filled by Operating System (OS) and storage applications will result in Data Unavailability (DU) at the storage system level. Our analysis also shows that Detectable Unrecoverable Errors (DUEs) on the cache data field are the major cause of DU in storage systems, while Silent Data Corruptions (SDCs) in the cache tag and data field are mainly the cause of DL in storage systems.

Improving Instruction TLB Reliability with Efficient Multi-bit Soft Error Protection

A Translation Lookaside Buffer (TLB) is a type of memory cache that is used to store recent translations of virtual to physical memory to reduce the access latency. Every time the processor accesses the virtual memory, it must be translated to the corresponding physical address, so the number of accesses to the TLB is high. Consequently, soft errors affecting the TLB can lead to hard fault, silent data corruption, and system freeze by corrupting its content. Many studies have proposed to provide protection for the Content Addressable Memory (CAM), which is a part of a TLB that stores the VPNs, but these protection techniques in most cases do not cover the case of multiple errors. This paper presents an efficient, fast and high error coverage approach to improve the reliability of TLB against Multiple Bit Upsets (MBUs) by considering the performance improvement with a low-cost overhead.

A Methodology to Assess Output Vulnerability Factors for Detecting Silent Data Corruption

As process technology scales, electronic devices become more susceptible to soft error induced by radiation. Silent data corruption (SDC) is considered the most severe outcome incurred by soft error. The effects of faulty variables on producing SDC vary widely. Without a profiling of vulnerability of variables, the derived detectors often incur low SDC detection rate or unacceptable overhead. To assess the vulnerability of variables to SDC, this paper proposes a metric called Output Vulnerability Factor (OVF). The metric is used to rank the variable's priority in the detector derivation process in order to selectively protect the most SDC-prone variable in the program. The calculation of OVF is based on enhanced Dynamic Dependence Graph (eDDG), a proposed instruction-level error propagation model. We filter out the edges representing the identified crash propagation path and perform a backward traversal of the eDDG to obtain SDC propagation path. Further, error masking probability is estimated for the edges refer to value comparison and logistic operation. Fault injections show that our approach achieves an SDC detection rate of 65.0% with the top 10% high OVF variables monitored. Compared with previous methods, the SDC detection rate increases by 12-21%.

Vulnerability Analysis of Instructions for SDC-Causing Error Detection

Due to the centralization of communication in the management of data generated by diverse Internet of Thing (IoT) devices, there is a lack of reliability when data is being transferred and stored. Among errors caused by various behaviors, Silent Data Corruption (SDC) error, owing to stealthy destruction without error prompt, is one of the most difficult data consistency problems in the storage system, whether it is a traditional multi-control, distributed storage, or public cloud storage. Nowadays, for SDC error detection, extracting instruction features to analyze vulnerabilities of programs or instructions has still not been fully explored. Literature generally just count the number of possible features, without mining the inter-characteristic of the instruction and correlation between them. Thus, we propose a method of SDC-causing Error Detection based on Support Vector Regression (SED-SVR) for fully exploiting the correlation between data features. Specifically, firstly, we extract instruction features based on the SDC vulnerability of program instructions by analyzing results of fault injections. Secondly, we establish the instruction SDC vulnerability prediction model based on SVR and propose our SED-SVR model. Thirdly, according to the predicted values of SDC vulnerability, we develop some solutions for faults tolerance of target programs by different granularity of instruction redundancy. The experimental results show that our SED-SVR has higher fault detection rate with lower performance overhead.

Identify Silent Data Corruption Vulnerable Instructions Using SVM

Silent data corruption (SDC) is the most insidious and harmful result type of soft error. Identify program vulnerable instructions (PVIns) that are likely to cause SDCs is extremely significant on selective software-based protection techniques. However, current identification techniques require tremendous fault injections or have non-negligible differences in performance among different programs as well as different program inputs. This paper proposes PVInsiden to reduce the cost of fault injection and improve the adaptability for programs and program inputs. Machine learning is used to learn a classifier which predicts whether an instruction is a PVIns. Partial fault injection is applied to generate a training dataset, reducing the cost of fault injection. The feature engineering, including selecting features and transforming the selected features into quantifiable representations is explored. Furthermore, the framework of learning the classifier is given. The experimental results show that PVInsiden only uses 35% fault injections to identify 85% PVIns with 80% precision, reducing the cost of fault injection efficiently. PVInsiden also shows high performance of precision, recall, and f0.5-score for different programs as well as different program inputs.

Generalized Numerical Entanglement for Reliable Linear, Sesquilinear and Bijective Operations on Integer Data Streams

We propose a new technique for the mitigation of fail-stop failures and/or silent data corruptions (SDCs) within linear, sesquilinear or bijective (LSB) operations on $M$ integer data streams ( $M\geq 3$ ). In the proposed approach, the $M$ input streams are linearly superimposed to form $M$ numerically entangled integer data streams that are stored in-place of the original inputs, i.e., no additional (aka. “checksum”) streams are used. An arbitrary number of LSB operations can then be performed in $M$ processing cores using these entangled data streams. The output results can be extracted from any $M-K$ entangled output streams by additions and arithmetic shifts, thereby mitigating $K$ fail-stop failures ( $K\leq \left\lfloor \frac{M-1}{2}\right\rfloor$ ), or detecting up to $K$ SDCs per $M$ -tuple of outputs at corresponding in-stream locations. Therefore, unlike other methods, the number of operations required for the entanglement, extraction and recovery of the results is linearly related to the number of the inputs and does not depend on the complexity of the performed LSB operations. Our proposal is validated within an Amazon EC2 instance (Haswell architecture with AVX2 support) via integer matrix product operations. Our analysis and experiments for fail-stop failure mitigation and SDC detection reveal that the proposed approach incurs 0.75 to 37.23 percent reduction in processing throughput in comparison to the equivalent error-intolerant processing. This overhead is found to be up to two orders of magnitude smaller than that of the equivalent checksum-based method, with increased gains offered as the complexity of the performed LSB operations is increasing. Therefore, our proposal can be used in distributed systems, unreliable multicore clusters and safety-critical applications, where robustness against failures and SDCs is a necessity.

On the Efficacy of ECC and the Benefits of FinFET Transistor Layout for GPU Reliability

Using error-correcting codes (ECCs) is considered one of the most effective ways to mask the effects of radiation-induced faults in memory and computing devices. Unfortunately, with the increased complexity of modern processors, there is a growing amount of hidden logic and memory resources, such as flip-flops in internal pipelines and queues, that cannot be easily protected by ECC. In this paper, we experimentally investigate the efficacy of using ECC to mask neutron-induced faults in modern graphics processing units (GPUs). In our analysis, we consider GPUs fabricated in CMOS and FinFET technologies. We show that changes in transistor technology can be as beneficial as using ECC for reducing silent data corruption rates. Finally, we compare fault-injection results, as carried out both on internal registers and at an instruction level, to better understand the effectiveness of ECC.

Resilient gossip-inspired all-reduce algorithms for high-performance computing: Potential, limitations, and open questions

We investigate the usefulness of gossip-based reduction algorithms in a high-performance computing (HPC) context. We compare them to state-of-the-art deterministic parallel reduction algorithms in terms of fault tolerance and resilience against silent data corruption (SDC) as well as in terms of performance and scalability. New gossip-based reduction algorithms are proposed, which significantly improve the state-of-the-art in terms of resilience against SDC. Moreover, a new gossip-inspired reduction algorithm is proposed, which promises a much more competitive runtime performance in an HPC context than classical gossip-based algorithms, in particular for low accuracy requirements.

Evaluating the Impact of the Instruction Set on Microprocessor Reliability to Soft Errors

Reliability in microprocessors is a concern. There are many techniques that add spatial and temporal redundancy, with the goal of increasing the error detection and correction capabilities of the system. In this paper, we present a methodology to estimate the intrinsic reliability of the instruction set architecture of a given processor. By analyzing the internal encoding of the instructions and their relations, an initial estimate of the silent data corruption rate that a certain architecture may suffer due to soft errors is obtained. The results provided by this methodology can be used to compare the intrinsic reliability of different processors and help decide the best protection technique in each case.

A Compiler Technique for Processor-Wide Protection From Soft Errors in Multithreaded Environments

Aggressive transistor scaling down and near-threshold computing have rendered modern microprocessor susceptible to soft errors. Software approaches that protect computations against soft errors are desirable because they offer flexible protection and are suitable for mixed-critical systems. In particular, fine-grain instruction duplication based techniques are deemed to be most effective; however, many of the existing instruction duplication techniques either suffer from many vulnerable intervals or are not suitable for multithreaded environments. In this paper, we present multithreded near zero silent data corruption (MZDC), a software scheme which provides high-level processor-wide error coverage in multithreaded environments. MZDC duplicates all programs' instructions and uses diagnosis block after replicated memory operations to overcome the inconsistency issue in a multithread environment. Statistical fault injection experiments on a dual-core ARM cortex-A53 $\mu$ architecturally simulated microprocessor show that on average, MZDC can achieve more than 37 $\times$ better fault coverage than the state-of-the-art.

D is CV ar

Aggressive technology scaling trends have made the hardware of high performance computing (HPC) systems more susceptible to faults. Some of these faults can lead to silent data corruption (SDC), and represent a serious problem because they alter the HPC simulation results. In this paper, we present a full-coverage, systematic methodology called D is CV ar to identify critical variables in HPC applications for protection against SDC. D is CV ar uses automatic differentiation (AD) to determine the sensitivity of the simulation output to errors in program variables. We empirically validate our approach in identifying vulnerable variables by comparing the results against a full-coverage code-level fault injection campaign. We find that our D is CV ar correctly identifies the variables that are critical to ensure application SDC resilience with a high degree of accuracy compared to the results of the fault injection campaign. Additionally, D is CV ar requires only two executions of the target program to generate results, whereas in our experiments we needed to perform millions of executions to get the same information from a fault injection campaign.