Large-scale High Performance Computing Research Articles

Exploring energy saving opportunities in fault tolerant HPC systems

Nowadays, improving the energy efficiency of high-performance computing (HPC) systems is one of the main drivers in scientific and technological research. As large-scale HPC systems require some fault-tolerant method, the opportunities to reduce energy consumption should be explored. In particular, rollback-recovery methods using uncoordinated checkpoints prevent all processes from re-executing when a failure occurs. In this context, it is possible to take actions to reduce the energy consumption of the nodes whose processes do not re-execute. This work is an extension of a previous one, in which we proposed a series of strategies to manage energy consumption at failure-time. In this work, we have enriched our simulator and the experimentation by including non-blocking communications (with and without system buffering) and a largest number of candidate processes to be analyzed. We have called the latter as cascade analysis, because it includes processes that gets blocked by communication indirectly with the failed process. The simulations show that the savings were negligible in the worst case, but in some scenarios, it was possible to achieve significant ones; the maximum saving achieved was 90% in a time interval of 16 minutes. As a result, we show the feasibility of improving energy efficiency in HPC systems in the presence of a failure.

Design and implementation of I/O performance prediction scheme on HPC systems through large-scale log analysis

Large-scale high performance computing (HPC) systems typically consist of many thousands of CPUs and storage units used by hundreds to thousands of users simultaneously. Applications from large numbers of users have diverse characteristics, such as varying computation, communication, memory, and I/O intensity. A good understanding of the performance characteristics of each user application is important for job scheduling and resource provisioning. Among these performance characteristics, I/O performance is becoming increasingly important as data sizes rapidly increase and large-scale applications, such as simulation and model training, are widely adopted. However, predicting I/O performance is difficult because I/O systems are shared among all users and involve many layers of software and hardware stack, including the application, network interconnect, operating system, file system, and storage devices. Furthermore, updates to these layers and changes in system management policy can significantly alter the I/O behavior of applications and the entire system. To improve the prediction of the I/O performance on HPC systems, we propose integrating information from several different system logs and developing a regression-based approach to predict the I/O performance. Our proposed scheme can dynamically select the most relevant features from the log entries using various feature selection algorithms and scoring functions, and can automatically select the regression algorithm with the best accuracy for the prediction task. The evaluation results show that our proposed scheme can predict the write performance with up to 90% prediction accuracy and the read performance with up to 99% prediction accuracy using the real logs from the Cori supercomputer system at NERSC.

Hyperparameter optimization of data-driven AI models on HPC systems

In the European Center of Excellence in Exascale Computing ”Research on AI- and Simulation-Based Engineering at Exascale” (CoE RAISE), researchers develop novel, scalable AI technologies towards Exascale. This work exercises High Performance Computing resources to perform large-scale hyperparameter optimization using distributed training on multiple compute nodes. This is part of RAISE’s work on data-driven use cases which leverages AI- and HPC cross-methods developed within the project. In response to the demand for parallelizable and resource efficient hyperparameter optimization methods, advanced hyperparameter search algorithms are benchmarked and compared. The evaluated algorithms, including Random Search, Hyperband and ASHA, are tested and compared in terms of both accuracy and accuracy per compute resources spent. As an example use case, a graph neural network model known as MLPF, developed for Machine Learned Particle-Flow reconstruction, acts as the base model for optimization. Results show that hyperparameter optimization significantly increased the performance of MLPF and that this would not have been possible without access to large-scale High Performance Computing resources. It is also shown that, in the case of MLPF, the ASHA algorithm in combination with Bayesian optimization gives the largest performance increase per compute resources spent out of the investigated algorithms.

Codesign for Extreme Heterogeneity: Integrating Custom Hardware With Commodity Computing Technology to Support Next-Generation HPC Converged Workloads

The future of high-performance computing (HPC) will be driven by the convergence of physical simulation, artificial intelligence, machine learning, and data science computing capabilities. While computational performance gains afforded by technology scaling, as predicted by Moore's Law, have enabled large-scale HPC system design and deployment using commodity CPU and GPU processing components, emerging technologies will be required to effectively support such converged workloads. These emerging technologies will integrate commodity computing components with custom processing and networking accelerators into ever-more heterogeneous architectures resulting in a diverse ecosystem of industry technology developers, university, and U.S. Government researchers. In this article, we describe efforts at the U.S. Department of Energy's Pacific Northwest National Laboratory to construct an end-to-end codesign framework that lays a groundwork for such an ecosystem, including notable outcomes, remaining challenges, and future opportunities.

I/O performance analysis of machine learning workloads on leadership scale supercomputer

The popularity of machine learning technologies and frameworks has led to an increasingly large number of machine learning workloads running on high-performance computing (HPC) clusters. The ML workflows are readily being adopted in diverse computational fields such as Biology, Physics, Materials, and Computer Science. The I/O behavior of the emerging ML workloads distinctly differs from the traditional HPC workloads, such as simulation or checkpoint/restart-based HPC I/O behavior. Additionally, the ML workloads have also pushed for the utilization of GPUs or a combination of CPUs and GPUs in addition to using only CPUs for computational tasks. The diverse and complex I/O behavior of ML workloads requires extensive study and is critical for the efficient performance of various layers of the I/O stack and the overall performance of HPC workloads. This work aims to fill the gap in understanding the I/O behavior of emerging ML workloads by providing an in-depth analysis of ML jobs running on large-scale leadership HPC systems. In particular, we have analyzed the behavior of jobs based on the scale of the jobs, the science domains, and the processing units used by the ML jobs. The analysis was performed on 23,000 ML jobs collected from one year of Darshan logs running on Summit, which is one of the fastest supercomputers. We also collect the CPU and GPU usage of 15,165 ML jobs by merging the Darshan dataset with the power usage of the processing units on Summit. Therefore, this paper is able to provide a systematic I/O characterization of ML workloads on a leadership scale HPC machine to understand how the I/O behavior differs for workloads across various science domains, the scale of workloads, and processing units and analyze the usage of parallel file system and burst buffer by ML I/O workloads. We have made several observations regarding I/O performances and access patterns through various analytical studies and discuss the important lessons learnt from the perspective of a ML user and a storage architect for emerging ML workloads running on large-scale supercomputers.

A scalability study of the Ice-sheet and Sea-level System Model (ISSM, version 4.18)

Abstract. Accurately modelling the contribution of Greenland and Antarctica to sea level rise requires solving partial differential equations at a high spatial resolution. In this paper, we discuss the scaling of the Ice-sheet and Sea-level System Model (ISSM) applied to the Greenland Ice Sheet with horizontal grid resolutions varying between 10 and 0.25 km. The model setup used as benchmark problem comprises a variety of modules with different levels of complexity and computational demands. The core builds the so-called stress balance module, which uses the higher-order approximation (or Blatter–Pattyn) of the Stokes equations, including free surface and ice-front evolution as well as thermodynamics in form of an enthalpy balance, and a mesh of linear prismatic finite elements, to compute the ice flow. We develop a detailed user-oriented, yet low-overhead, performance instrumentation tailored to the requirements of Earth system models and run scaling tests up to 6144 Message Passing Interface (MPI) processes. The results show that the computation of the Greenland model scales overall well up to 3072 MPI processes but is eventually slowed down by matrix assembly, the output handling and lower-dimensional problems that employ lower numbers of unknowns per MPI process. We also discuss improvements of the scaling and identify further improvements needed for climate research. The instrumented version of ISSM thus not only identifies potential performance bottlenecks that were not present at lower core counts but also provides the capability to continually monitor the performance of ISSM code basis. This is of long-term significance as the overall performance of ISSM model depends on the subtle interplay between algorithms, their implementation, underlying libraries, compilers, runtime systems and hardware characteristics, all of which are in a constant state of flux. We believe that future large-scale high-performance computing (HPC) systems will continue to employ the MPI-based programming paradigm on the road to exascale. Our scaling study pertains to a particular modelling setup available within ISSM and does not address accelerator techniques such as the use of vector units or GPUs. However, with 6144 MPI processes, we identified issues that need to be addressed in order to improve the ability of the ISSM code base to take advantage of upcoming systems that will require scaling to even higher numbers of MPI processes.

Using HPC infrastructures for deep learning applications in fusion research

In the fusion community, the use of high performance computing (HPC) has been mostly dominated by heavy-duty plasma simulations, such as those based on particle-in-cell and gyrokinetic codes. However, there has been a growing interest in applying machine learning for knowledge discovery on top of large amounts of experimental data collected from fusion devices. In particular, deep learning models are especially hungry for accelerated hardware, such as graphics processing units (GPUs), and it is becoming more common to find those models competing for the same resources that are used by simulation codes, which can be either CPU- or GPU-bound. In this paper, we give examples of deep learning models—such as convolutional neural networks, recurrent neural networks, and variational autoencoders—hat can be used for a variety of tasks, including image processing, disruption prediction, and anomaly detection on diagnostics data. In this context, we discuss how deep learning can go from using a single GPU on a single node to using multiple GPUs across multiple nodes in a large-scale HPC infrastructure.

Artificial Intelligence and Radiomics in Head and Neck Cancer Care: Opportunities, Mechanics, and Challenges.

The advent of large-scale high-performance computing has allowed the development of machine-learning techniques in oncologic applications. Among these, there has been substantial growth in radiomics (machine-learning texture analysis of images) and artificial intelligence (which uses deep-learning techniques for "learning algorithms"); however, clinical implementation has yet to be realized at scale. To improve implementation, opportunities, mechanics, and challenges, models of imaging-enabled artificial intelligence approaches need to be understood by clinicians who make the treatment decisions. This article aims to convey the basic conceptual premises of radiomics and artificial intelligence using head and neck cancer as a use case. This educational overview focuses on approaches for head and neck oncology imaging, detailing current research efforts and challenges to implementation.

Study of interconnect errors, network congestion, and applications characteristics for throttle prediction on a large scale HPC system

Today’s High Performance Computing (HPC) systems contain thousand of nodes which work together to provide performance in the order of petaflops. The performance of these systems depends on various components like processors, memory, and interconnect. Among all, interconnect plays a major role as it glues together all the hardware components in an HPC system. A slow interconnect can impact a scientific application running on multiple processes severely as they rely on fast network messages to communicate and synchronize frequently. Unfortunately, the HPC community lacks a study that explores different interconnect errors, congestion events and applications characteristics on a large-scale HPC system. In our previous work, we process and analyze interconnect data of the Titan supercomputer to develop a thorough understanding of interconnects faults, errors, and congestion events. In this work, we first show how congestion events can impact application performance. We then investigate application characteristics interaction with interconnect errors and network congestion to predict applications encountering congestion with more than 90% accuracy.

Toward lower-diameter large-scale HPC and data center networks with co-packaged optics

We investigate the advantages of using co-packaged optics for building low-diameter, large-scale high-performance computing (HPC) and data center networks. The increased escape bandwidth offered by co-packaged optics can enable high-radix switch implementations of more than 150 switch ports, which can be combined with data rates of up to 400 Gb/s per port. From the network architecture perspective, the key benefits of using co-packaged optics in future fat-tree networks include (a) the ability to implement large-scale topologies of > --> 11 , 000 end points by eliminating the need for a third switching layer and (b) the ability to provide up to 4 × higher bisection bandwidth compared to existing solutions, reducing at the same time the number of required switch application-specific integrated circuits by > --> 80 % . From the network operation perspective, both reduced energy consumption and lower packet delays can be achieved since fewer hops are required; i.e., packets need to traverse fewer serializer/deserializer lanes and fewer switch buffers, which reduces the probability of contending with other packets and improves the tolerance of network congestion. The performance of the proposed architecture is evaluated via discrete-event simulations for a wide range of representative HPC synthetic-traffic cases that include both hotspot and non-hotspot scenarios. The simulation results suggest that co-packaged optics form a promising solution to keep up with bandwidth scaling in future networks, while the reduced number of switching layers can lead to significant mean packet delay improvements that start from 30% and reach up to 74% for high-load conditions.

High performance computing of DGDFT for tens of thousands of atoms using millions of cores on Sunway TaihuLight

High performance computing (HPC) is a powerful tool to accelerate the Kohn–Sham density functional theory (KS-DFT) calculations on modern heterogeneous supercomputers. Here, we describe a massively parallel implementation of discontinuous Galerkin density functional theory (DGDFT) method on the Sunway TaihuLight supercomputer. The DGDFT method uses the adaptive local basis (ALB) functions generated on-the-fly during the self-consistent field (SCF) iteration to solve the KS equations with high precision comparable to plane-wave basis set. In particular, the DGDFT method adopts a two-level parallelization strategy that deals with various types of data distribution, task scheduling, and data communication schemes, and combines with the master–slave multi-thread heterogeneous parallelism of SW26010 processor, resulting in large-scale HPC KS-DFT calculations on the Sunway TaihuLight supercomputer. We show that the DGDFT method can scale up to 8,519,680 processing cores (131,072 core groups) on the Sunway TaihuLight supercomputer for studying the electronic structures of two-dimensional (2D) metallic graphene systems that contain tens of thousands of carbon atoms.

Large-scale HPC deployment of Scalable CyberInfrastructure for Artificial Intelligence and Likelihood Free Inference (SCAILFIN)

The NSF-funded Scalable CyberInfrastructure for Artificial Intelligence and Likelihood Free Inference (SCAILFIN) project aims to develop and deploy artificial intelligence (AI) and likelihood-free inference (LFI) techniques and software using scalable cyberinfrastructure (CI) built on top of existing CI elements. Specifically, the project has extended the CERN-based REANA framework, a cloud-based data analysis platform deployed on top of Kubernetes clusters that was originally designed to enable analysis reusability and reproducibility. REANA is capable of orchestrating extremely complicated multi-step workflows, and uses Kubernetes clusters both for scheduling and distributing container-based workloads across a cluster of available machines, as well as instantiating and monitoring the concrete workloads themselves. This work describes the challenges and development efforts involved in extending REANA and the components that were developed in order to enable large scale deployment on High Performance Computing (HPC) resources. Using the Virtual Clusters for Community Computation (VC3) infrastructure as a starting point, we implemented REANA to work with a number of differing workload managers, including both high performance and high throughput, while simultaneously removing REANA’s dependence on Kubernetes support at the workers level.

CYBELE – Fostering precision agriculture & livestock farming through secure access to large-scale HPC enabled virtual industrial experimentation environments fostering scalable big data analytics

According to McKinsey & Company, about a third of food produced is lost or wasted every year, amounting to a $940 billion economic hit. Inefficiencies in planting, harvesting, water use, reduced animal contributions, as well as uncertainty about weather, pests, consumer demand and other intangibles contribute to the loss. Precision Agriculture (PA) and Precision Livestock Farming (PLF) come to assist in optimizing agricultural and livestock production and minimizing the wastes and costs aforementioned. PA is a technology-enabled, data-driven approach to farming management that observes, measures, and analyzes the needs of individual fields and crops. PLF is also a technology-enabled, data-driven approach to livestock production management, which exploits technology to quantitatively measure the behavior, health and performance of animals. Big data delivered by a plethora of data sources related to these domains, has a multitude of payoffs including precision monitoring of fertilizer and fungicide levels to optimize crop yields, risk mitigation that results from monitoring when temperature and humidity levels reach dangerous levels for crops, increasing livestock production while minimizing the environmental footprint of livestock farming, ensuring high levels of welfare and health for animals, and more. By adding analytics to these sensor and image data, opportunities also exist to further optimize PA and PLF by having continuous data on how a field or the livestock is responding to a protocol. For these domains, two main challenges exist: 1) to exploit this multitude of data facilitating dedicated improvements in performance, and 2) to make available advanced infrastructure so as to harness the power of this information in order to benefit from the new insights, practices and products, efficiently time-wise, lowering responsiveness down to seconds so as to cater for time-critical decisions. The current paper aims to introduce CYBELE, a platform aspiring to safeguard that the stakeholders involved in the agri-food value chain (research community, SMEs, entrepreneurs, etc.) have integrated, unmediated access to a vast amount of very large scale datasets of diverse types and coming from a variety of sources, and that they are capable of actually generating value and extracting insights out of these data, by providing secure and unmediated access to large-scale High Performance Computing (HPC) infrastructures supporting advanced data discovery, processing, combination and visualization services, solving computationally-intensive challenges modelled as mathematical algorithms requiring very high computing power and capability.

Robust Identification of Thermal Models for In-Production High-Performance-Computing Clusters With Machine Learning-Based Data Selection

Power and thermal management are critical components of high-performance-computing (HPC) systems, due to their high-power density and large total power consumption. The assessment of thermal dissipation by means of compact models directly from the thermal response of the final device enables more robust and precise thermal control strategies as well as automated diagnosis. However, when dealing with large-scale systems “in production,” the accuracy of learned thermal models depends on the dynamics of the power excitation, which depends also on the executed workload, and measurement nonidealities such as quantization. In this article we show that, using an advanced system identification algorithm, we are able to generate very accurate thermal models (average error lower than our sensors quantization step of 1 °C) for a large-scale HPC system on real workloads for very long time periods. However, we also show that: 1) not all real workloads allow for the identification of a good model and 2) starting from the theory of system identification it is very difficult to evaluate if a trace of data leads to a good estimated model. We then propose and validate a set of techniques based on machine learning and deep learning algorithms for the choice of data traces to be used for model identification. We also show that deep learning techniques are absolutely necessary to correctly choose such traces up to 96% of the times.

A Realtime Monitoring Method for Cluster System Running State Based on Network

High Performance Computing (HPC) system running state monitoring is an important task. Small-scale cluster systems exist separately as one whole system or exist as a part of a large-scale HPC system. Such cluster systems are often the core of the entire system. Therefore, for such systems, there should be more timely and accurate monitoring of running state. However, various HPC system monitoring softwares currently focus on the aggregation of overall running and resource state, and cannot provide realtime monitoring, detailed information, and process control. Based on the existing single-machine real-time monitoring tools, this paper adds network-based information exchange protocol, multi-threaded parallel information acquisition service, extends information representation and display output style, adds remote process control, then carries out related tests, which implements a more efficient real-time running state monitoring tool for small-scale cluster systems.

High performance computing simulations of spall phenomenon in a submicron thick nanocrystalline aluminum

The spall failure phenomenon is investigated at nano-scale in a plate-on-plate impact configuration using large-scale molecular dynamics (MD) high performance computing (HPC) simulationson three multi-billion 0.5 μm thick nanocrystalline aluminum (nc-Al) systems, each with an average grain size in the range of 18–100 nm and each under impact velocities in the range of 0.7–1.5 km s−1. Using a material conserving atom section-averaging process, distributions of several mechanical responses were obtained and the extremely transient spall failure phenomena were located in the interaction zones of the stress release waves released from the impacted and free ends of the atom systems. The spall zones’ thicknesses were estimated along with the spall strengths. For the nc-Al atom systems, the spall strength was observed to increase as the impact velocity increased from 0.7 to 1.5 km s−1, but only showed a slight decrease as the grain size increased from 18 to 100 nm. The spall strengths thus estimated from the stress release waves’ interaction zones were found to be conservative when compared to the traditional estimates obtained from the pullback velocity signatures in the free-end velocity evolutions. The MD results were further analyzed using a crystal analysis algorithm and a twin dislocation identification method to obtain the densities of the atomistic microstructures evolving under the interaction of the stress release waves. High-fidelity large-scale HPC simulation results showed that certain dislocation partials strongly influenced the spall response. The Stair-rod type dislocation partials increased by more than a factor of 10 during the interaction of the stress release waves as the spall failure commenced.

Resilience-Aware Resource Management for Exascale Computing Systems

With the increases in complexity and number of nodes in large-scale high performance computing (HPC) systems over time, the probability of applications experiencing runtime failures has increased significantly. Projections indicate that exascale-sized systems are likely to operate with mean time between failures (MTBF) of as little as a few minutes. Several strategies have been proposed in recent years for enabling systems of these extreme sizes to be resilient against failures. This work provides a comparison of four state-of-the-art HPC resilience protocols that are being considered for use in exascale systems. We explore the behavior of each resilience protocol operating under the simulated execution of a diverse set of applications and study the performance degradation that a large-scale system experiences from the overhead associated with each resilience protocol as well as the re-computation needed to recover when a failure occurs. Using the results from these analyses, we examine how resource management on exascale systems can be improved by allowing the system to select the optimal resilience protocol depending upon each application's execution characteristics, as well as providing the system resource manager the ability to make scheduling decisions that are “resilience aware” through the use of more accurate execution time predictions.

HPC simulations of shock front evolution for a study of the shock precursor decay in a submicron thick nanocrystalline aluminum

The Hugoniot elastic limit (HEL, or the shock precursor) decay phenomenon was investigated under an uniaxial strain condition, in a plate-on-plate impact configuration, using large-scale molecular dynamics (MD) high performance computing (HPC) simulations on a multi-billion 5000 Å thick nanocrystalline aluminum (nc-Al) system with an average grain size of 1000 Å and at five impact velocities ranging from 0.7 to 1.5 km s−1. The averaged stress and strain distributions were obtained in the shock fronts’ travel direction using a material conserving atom slicing method. The loading paths in terms of the Rayleigh lines experienced by the atom system in the evolving shock fronts exhibited a strong dependency on the shock stress levels. This dependency decreased as the impact velocity increased from 0.7 to 1.5 km s−1. By combining the HELs from MD results with plate impact experimental data, the precursor decay for the nc-Al was predicted from nano-to-macro scale thickness range. The evolving shock fronts were characterized in terms of parameters such as the shock front thickness, shock rise time and strain rate. The MD results were further analyzed using a crystal analysis algorithm and a twin dislocation identification method to obtain the densities of the atomistic defects evolving behind the evolving shock fronts. High-fidelity large-scale HPC simulation results showed that certain dislocation partials strongly influenced the elastic–plastic transition response across the HELs. The twinning dislocations increased by more than a factor of 10 during the transition and remained constant under further shock compression.

Designing scalable FPGA architectures using high-level synthesis

Massive spatial parallelism at low energy gives FPGAs the potential to be core components in large scale high performance computing (HPC) systems. In this paper we present four major design steps that harness high-level synthesis (HLS) to implement scalable spatial FPGA algorithms. To aid productivity, we introduce the open source library hlslib to complement HLS. We evaluate kernels designed with our approach on an FPGA accelerator board, demonstrating high performance and board utilization with enhanced programmer productivity. By following our guidelines, programmers can use HLS to develop efficient parallel algorithms for FPGA, scaling their implementations with increased resources on future hardware.

Trends in Data Locality Abstractions for HPC Systems

The cost of data movement has always been an important concern in high performance computing (HPC) systems. It has now become the dominant factor in terms of both energy consumption and performance. Support for expression of data locality has been explored in the past, but those efforts have had only modest success in being adopted in HPC applications for various reasons. them However, with the increasing complexity of the memory hierarchy and higher parallelism in emerging HPC systems, locality management has acquired a new urgency. Developers can no longer limit themselves to low-level solutions and ignore the potential for productivity and performance portability obtained by using locality abstractions. Fortunately, the trend emerging in recent literature on the topic alleviates many of the concerns that got in the way of their adoption by application developers. Data locality abstractions are available in the forms of libraries, data structures, languages and runtime systems; a common theme is increasing productivity without sacrificing performance. This paper examines these trends and identifies commonalities that can combine various locality concepts to develop a comprehensive approach to expressing and managing data locality on future large-scale high-performance computing systems.