Introducing a Reliability Analysis Framework for High Performance Computing Environments

Abstract

Supercomputing environments are becoming the norm for daily use. However, their complex infrastructure makes troubleshooting and monitoring failures extremely difficult. This is because these infrastructures contain thousands of nodes representing various applications and processors. To address these concerns, we propose a real-time reliability analysis framework for high performance computing (HPC) environments where the contributions are three-fold. First, an improved data network extrapolation (DNE) methodology is proposed as a pre-processing module. This component incorporates the system failure information (i.e. job, fault, and error log files) and performs robust job-based failure accounting for sequential and parallel jobs. This element also performs cross-referencing to compute task-based failure accounting, where the assumption is made that tasks are comprised of either one or more jobs. Next, a reliability characterization and analysis (RCA) schema is proposed that takes the failure information from the DNE process to perform survival analyses on each individual node in addition to the entire reliability infrastructure. This is coupled with a failure metrics characterization (FMC) schema that estimates the failure metrics such as the mean time to failure (MTTF) as well as the hazard rate. Additionally, a comparative analysis is made between the Log-Normal or Weibull distributions in terms of modeling job and task-based failure activity. Empirical analysis using the Structural Simulation Toolkit (SST) illustrate the promise of this approach in terms of characterizing, monitoring, and troubleshooting failure behavior. The results of this work can aide systems administrators the dynamic tools to pinpoint and monitor failure behavior; its impacts; and alternative job-scheduling policies without interrupting production processes.