System-wide Outages Research Articles

Data-Driven Application-Oriented Reliability Model of a High-Performance Computing System

Reliability analysis and performance evaluation are complementary methods to quantify nonfunctional aspects of a system. However, a range of factors such as concurrency and heterogeneity quickly exacerbate the state-space explosion problem when attempting detailed system-level modeling and simulation of high-performance computing (HPC) systems. To overcome these impediments to modeling and analysis, this article develops a hierarchical model of an application that implements checkpointing running in an HPC environment subject to application, network, and system-wide outages. The modeling approach ensures that the number of states is linear in the number of checkpoints and possesses a low constant factor for the number of recovery states most relevant to the external influences contributing to degraded application performance. We illustrate the types of analysis enabled by the model through a series of examples with parameters determined empirically from data logs of the Blue Waters supercomputer located at the University of Illinois at Urbana–Champaign. A comprehensive comparative analysis of the model parameters indicates that lowering the failure rate of network nodes would most significantly reduce application downtime. We also discuss how the modeling approach can be used to objectively assess both current and hypothetical future systems to identify competitive designs and enhancements.

Parallelization and I/O Performance Optimization of a Global Nonhydrostatic Dynamical Core using MPI

The Global ‐ Regional Integrated forecast SysTem (GRIST) is the next- generation weather and climate integrated model dynamic framework developed by Chinese Academy of Meteorological Sciences. In this paper, we present several changes made to the global nonhydrostatic dynamical (GND) core, which is part of the ongoing prototype of GRIST. The changes leveraging MPI and PnetCDF techniques were targeted at the parallelization and performance optimization to the original serial GND core. Meanwhile, some sophisticated data structures and interfaces were designed to adjust flexibly the size of boundary and halo domains according to the variable accuracy in parallel context. In addition, the I/O performance of PnetCDF decreases as the number of MPI processes increases in our experimental environment. Especially when the number exceeds 6000, it caused system-wide outages (SWO). Thus, a grouping solution was proposed to overcome that issue. Several experiments were carried out on the supercomputing platform based on Intel x86 CPUs in the National Supercomputing Center in Wuxi. The results demonstrated that the parallel GND core based on grouping solution achieves good strong scalability and improves the performance significantly, as well as avoiding the SWOs.

A fuzzy based parametric monitoring and control algorithm for distinctive loads to enhance the stability in rural islanded microgrids

Effective monitoring and control of isolated rural microgrid in the developing world is challenging. The modern communication and monitoring is difficult to handle in such communities due to a complicated approach to the area, lack of modern facilities and unavailability of skilled manpower. Implementation of a microgrid in such areas using intermittent renewable sources and limited storage is challenging. Uncontrolled load consumption leads to the system-wide outages due to prolonged storage utilization in peak hours and is referred here as battery storage stress hours (BSSH). This research is focused to study and analyze the behavior of parametric load monitoring and control algorithm that could control the distinctive load of the microgrid during BSSH. In the proposed algorithm, the residential loads are distinctively controlled while utilizing the three locally available parameters that are the state of the charge of storage, solar irradiations and ambient temperature. In other words, the natural parameter variations have been uniquely utilized as a monitoring tool for load control. The fuzzy controller takes a decision for the activation or deactivation of any load based on the three parameters variation ranges. It is observed from the simulation and experimental results that while only utilizing locally available parameters the effective load control is possible.

Integration of renewable distributed generation with storage and demand side load management in rural islanded microgrid

The integration between distributed generations, storage, and load for islanded microgrids located in rural areas of developing countries offer serious challenges. The rural microgrid, catering the energy needs of rural communities, has no access to modern information and communication amenities like internet, communication architectures, and backup power. The lack of information and communication technologies (ICT) and backup power sources in rural microgrid lead to uncontrolled load consumption and inefficient storage dispatch resulting in system-wide outages. This paper proposes a novel approach to overcome challenges such as affordability and accessibility. A proactive load controlling and prioritized smart load curtailment is proposed in real time using local system parameters and human activities. The objective is to reduce blackouts and ensure maximum load supply without using conventional backup sources, computational forecasting, scheduling tools, and sophisticated communication architectures. The fuzzy logic controller is proposed to proactively decide for a percentage of load to be curtailed. Human Activities Tracking System (HATS) for smart load curtailment is used for balanced and efficient integration between the distributed generation, storage, and load. Four system parameters, storage drain time, state of charge (SoC), running load, and solar irradiations, are used as input to the fuzzy controller. HATS controller is used to smartly prioritize the load curtailment based on human occupancy in different buildings. Human activities are monitored using pyroelectric sensors installed at the entrance of different buildings. The performance of the system has been assessed while comparing the proposed methodology, the SoC controlling method (SCM). While using the proposed methodology, the blackout was drastically reduced to 96 to 100%. State of charge depletion rate was considerably less than conventional methods.

Resiliency of HPC Interconnects: A Case Study of Interconnect Failures and Recovery in Blue Waters

Availability of the interconnection network in high-performance computing (HPC) systems is fundamental to sustaining the continuous execution of applications at scale. When failures occur, interconnect recovery mechanisms orchestrate complex operations to recover network connectivity between the nodes. As the scale and design complexity of HPC systems increase, so does the system's susceptibility to failures during execution of interconnect-recovery procedures. This study characterizes the recovery procedures of the Gemini interconnect network, the largest Gemini network built by Cray, on Blue Waters, a 13.3 petaflop supercomputer at the National Center for Supercomputing Applications (NCSA). We propose a propagation model that captures interconnect failures and recovery procedures to help understand types of failures and their propagation in both the system and applications during recovery. The measurements show that recovery procedures occur very frequently and that the unsuccessful execution of recovery procedures, when additional failures occur during recovery, causes system-wide outages (SWOs, 28 out of 101) and application failures (3.4 percent of all running applications).

A Predictive Energy Management System Using Pre-Emptive Load Shedding for Islanded Photovoltaic Microgrids

This paper presents an energy management system (EMS) for an islanded microgrid with photovoltaic generation and battery storage. The system uses a predictive approach to set operational schedules in order to minimize system-wide outages in the microgrid, specifically through pre-emptive load shedding. Four-times daily updated online weather forecasts are combined with the photovoltaic system model to predict energy production over a 48 h period. These predictions are used, along with load forecasts and a model of the energy storage system, to predict the state-of-charge and characterize potential upcoming outages. Outage mitigation actions using pre-emptive load shedding are then planned and executed to avoid outages or minimize the duration of unavoidable outages. The approach also features bounds on the battery state-of-charge to account for uncertainties in the estimate of the stored energy. The EMS has been implemented using an event-driven framework with TCP/IP communication, which is modular and extensible to more complex system configurations. The approach has been validated through simulations and experiments, which demonstrate its feasibility and potential, for the chosen test scenario, to reduce the outage duration by 87% to 100%.

Metric selection and anomaly detection for cloud operations using log and metric correlation analysis

Cloud computing systems provide the facilities to make application services resilient against failures of individual computing resources. However, resiliency is typically limited by a cloud consumer’s use and operation of cloud resources. In particular, system operations have been reported as one of the leading causes of system-wide outages. This applies specifically to DevOps operations, such as backup, redeployment, upgrade, customized scaling, and migration – which are executed at much higher frequencies now than a decade ago. We address this problem by proposing a novel approach to detect errors in the execution of these kinds of operations, in particular for rolling upgrade operations. Our regression-based approach leverages the correlation between operations’ activity logs and the effect of operation activities on cloud resources. First, we present a metric selection approach based on regression analysis. Second, the output of a regression model of selected metrics is used to derive assertion specifications, which can be used for runtime verification of running operations. We have conducted a set of experiments with different configurations of an upgrade operation on Amazon Web Services, with and without randomly injected faults to demonstrate the utility of our new approach.

Fault-tolerant performance checking application for distributed computing and supply chain networks

This paper presents a model/simulation of a fault tolerant system along with its implementation. The system also keeps track on the performance of the system and adjust it when a need arises. In order for the system to be fault tolerant we have to provide redundancies in the system. This is extremely important for systems, such as supply chain networks, whereby agents communicate with each other and a problem in the system can induce a ripple effect, leading to system-wide outage. Along with speed/bandwidth a need arises for this data flow to be consistent and the mechanism to be fault tolerant. The architecture is based on the concept of grid computing, but in this paper we focus mainly on developing a simulation of the process and develop components, which can monitor the simulation along with providing fault tolerance. Monitoring and fault tolerance are related to each other, as we can't provide fault tolerance without monitoring the system.

SIR-based call admission control for DS-CDMA cellular systems

Signal-to-interference ratio (SIR)-based call admission control (CAC) algorithms are proposed and studied in a DS-CDMA cellular system. Residual capacity is introduced as the additional number of initial calls a base station can accept such that system-wide outage probability will be guaranteed to remain below a certain level. The residual capacity at each cell is updated dynamically according to the reverse-link SIR measurements at the base station. A 2/sup k/ factorial experimental design and analysis via computer simulations is used to study the impact of the parameters used in the algorithms. The influence of these parameters on system performance, namely blocking probability and outage probability, is then examined via simulation. The performance of the algorithms is compared together with that of a fixed call admission control scheme (fixed CAC) under both homogeneous and hot spot traffic loading. The results show that SIR-based CAC always outperforms fixed CAC even under overload situations, which is not the case in FDMA/TDMA cellular systems. The primary benefit of SIR-based CAC in DS-CDMA cellular systems, however, lies in improving the system performance under hot spot traffic.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>