High Performance Computing Data Centers Research Articles

District heating utilizing waste heat of a data center: High-temperature heat pumps

Data centers are energy-intensive facilities with substantial low-grade waste heat. High-temperature heat pumps can be critical in boosting the data center’s waste heat for district heating, improving the system-level energy efficiency of data centers, and reducing CO2 emissions in district heating. This study built thermodynamic models to assess high-temperature heat pumps with six configurations using low global warming potential refrigerants to supply heat up to 120 °C. The heat pump configurations include single-stage or two-stage cycles with advanced components, such as internal heat exchanger, economizer, flash tank, or parallel compressor. The refrigerants include R1234ze(Z), R1233ed(E), R1224yd(Z), R600, and R600a, and R245fa is used as a reference. A case study was carried out to recover the waste heat from the Frontier high-performance computing data center and provide hot water for district heating at the US Department of Energy’s Oak Ridge National Laboratory campus. The optimized performance of high-temperature heat pumps is characterized with various effectiveness of internal heat exchangers, and the operating parameters of economizer or flash tank, as well as their combination. The results show that the configurations of two-stage cycles with internal heat exchanger + flash tank and internal heat exchanger + economizer/parallel-compressor provide the highest coefficient of performance under scenarios of the maximum allowable value and a fixed value (0.3) of the internal heat exchangers’ effectiveness, respectively. R1234ze(Z) and R600a are the most promising refrigerants, considering trade-offs between the coefficient of performance and the volumetric heating capacity. The single-stage cycle with internal heat exchanger + economizer/parallel-compressor using R1234ze(Z) is recommended for utilizing Fronter’s waste heat in district heating. A one mega-watt high-temperature heat pump will reduce 33,100–33,200 metric tons of CO2 emission annually, corresponding to 85.4 %–85.6 % of equivalent CO2 emissions from natural gas boilers. This study provides good guidelines for designing and deploying high-temperature heat pumps to support sustainable data centers and decarbonize district heating in the US.

Energy-focused simulation of edge computing architectures in 5G networks

While cloud computing is crucial in processing data from devices with low computational power, the latency introduced by the Internet backhaul limits real-time applications. By situating computing resources at the network’s edge, edge computing offers low-latency services by offloading computations from high-performance computing (HPC) data centers to the edge servers, reducing wide Area network (WAN) strain. As a result, edge computing has unlocked opportunities for innovative applications that were previously unfeasible, such as connected vehicles or medical robotics. Nonetheless, deploying the infrastructure required to support edge computing services raises sustainability and energy consumption concerns. Consequently, the development of tools enabling researchers to explore innovative approaches to reducing the energy impact of edge computing is crucial. In this work, we present MintEDGE, a network simulator focused on the energy consumption of edge computing. Our simulator allows testing energy-saving approaches and task placement algorithms in realistic large-scale scenarios encompassing entire regions.

DAG-based workflows scheduling using Actor–Critic Deep Reinforcement Learning

High-Performance Computing (HPC) is essential to support the advance in multiple research and industrial fields. Despite the recent growth in processing and networking power, the HPC Data Centers (DCs) are finite, and should be carefully managed to host multiple jobs. The scheduling of tasks (composing a job) is a crucial and complex task, once the reflexes of the scheduler’s decisions are perceptible both for users (e.g., slowdown) and for infrastructure administrators (e.g., use of resources and queue length). In fact, the process of scheduling workflows atop a DC can be modeled as a graph mapping problem. While an undirected graph is used to represent the DC, a Directed Acyclic Graph (DAG) is used to express the tasks dependencies. Each vertex and edge from both graphs can have weights associated with them, denoting the residual capacities for DC resources, as well as computing and networking demands for workflows. Motivated by the combinatorial explosion of the aforementioned scheduling problem, the integration of Machine Learning (ML) for generating or improving scheduling policies is a reality, however the proposals in the specialized literature opt, mostly, for using simplified models to reduce the search space or are trained to specific scenarios, which leads to policies that eventually fall short of real DCs expectations. Given this challenge, this work applies Actor–Critic (AC) Reinforcement Learning (RL) to schedule DAG-based workflows. Instead of proposing a new policy, the AC RL is used to select the appropriated scheduling policy from a pool of consolidated algorithms, guided by the DAGs workload and DC usage. The AC RL-based scheduler analyzes the DAGs queue and the DC status to define which algorithms are better suited to improve the overall performance indicators in each scenario instance. The simulation protocol comprises multiple analysis with distinct workload configurations, number of jobs, queue ordering polices and strategies to select the target DC servers. The results demonstrated that the AC RL selects the scheduling policy which fits the current workload and DC status.

O-Star: An Optical Switching Architecture Featuring Mode and Wavelength-Division Multiplexing for On-Chip Many-Core Systems

In this paper, we propose O-Star, a scalable optical switching architecture for on-chip many-core systems, employing hybrid mode and wavelength division multiplexing technology. The O-Star uses the Benes topology as the core switching module to realize non-blocking switching. Besides, we design a wavelength and mode allocation module to enable each processor to transmit data in parallel. To quantitatively analyze the minimum hardware cost required, we establish a mathematical model for the different number of processors. As a proof of concept, a 64-core optical switching architecture featuring 8-port Benes topology, 2-mode, and 4-wavelength channels is simulated with single channel 25 Gbps data rate. The O-Star is flexible to scale, both for the number of supported processors and the switching capacity. O-Star holds promise for realizing large-scale optical switching networks to address the incoming challenges of high-performance computing (HPC) systems and data center networks (DCNs).

CPU Overclocking: A Performance Assessment of Air, Cold Plates, and Two-Phase Immersion Cooling

Computing capacity has always been on the upward climb due to the constant technological improvements in semiconductor manufacturing and packaging industry. This growth in computing capability is usually accompanied by a steep rise in heat flux density associated with the electronic component (CPUs or GPUs, for example). High-performance computing (HPC) data centers often employ several of these high-performance devices for crunching their enormous artificial intelligence (AI) or scientific computing workloads. State-of-the-art air cooling technologies throttle after a certain heat flux level and would require bigger heat sinks driving enormous airflow through it, which may not be desirable from a data center operation standpoint. Hence, a lookout for advanced thermal management techniques is quite imperative. In this article, a commercially available intel overclockable CPU i9-9900k was exercised with high performance workloads while employing and evaluating three different cooling technologies namely air-cooled, cold plates, and two-phase (2P) immersion. The high heat carrying capacity associated with cold plates and 2P immersion techniques outperformed the air-cooled solution by constantly yielding higher CPU clock rates up to 41% utilizing cold plates and 51% employing 2P immersion. The performance of cold plates and 2P immersion was evaluated at different functional points (different coolant operating temperatures and 2P coolant types for example) to understand how the technologies would respond in a legacy or modern data center operation. It was again observed that both 2P immersion technology and cold plates provided the least thermal resistance (as low as 0.247 °C/W) path to heat transfer and therefore provided higher computational performance and efficiency compared to air cooling. Further improvement in performance is not limited by cooling but by packaging constraints. Advanced packaging techniques coupled with minimizing the interfacial resistance can reap enhanced performance and energy efficiency gains using these modes of cooling. This article demonstrates the potential increase in virtual machine (VM) performance that can be attained using cold plates or 2P immersion. Authors recommend adoption of liquid cooling in early chip design and architecture phases. Finally, the article provides a generic overclocking methodology that the industry can use to qualify postturbo server class chip performance.

Intra- and Inter-Server Smart Task Scheduling for Profit and Energy Optimization of HPC Data Centers

Servers in a data center are underutilized due to over-provisioning, which contributes heavily toward the high-power consumption of the data centers. Recent research in optimizing the energy consumption of High Performance Computing (HPC) data centers mostly focuses on consolidation of Virtual Machines (VMs) and using dynamic voltage and frequency scaling (DVFS). These approaches are inherently hardware-based, are frequently unique to individual systems, and often use simulation due to lack of access to HPC data centers. Other approaches require profiling information on the jobs in the HPC system to be available before run-time. In this paper, we propose a reinforcement learning based approach, which jointly optimizes profit and energy in the allocation of jobs to available resources, without the need for such prior information. The approach is implemented in a software scheduler used to allocate real applications from the Princeton Application Repository for Shared-Memory Computers (PARSEC) benchmark suite to a number of hardware nodes realized with Odroid-XU3 boards. Experiments show that the proposed approach increases the profit earned by 40% while simultaneously reducing energy consumption by 20% when compared to a heuristic-based approach. We also present a network-aware server consolidation algorithm called Bandwidth-Constrained Consolidation (BCC), for HPC data centers which can address the under-utilization problem of the servers. Our experiments show that the BCC consolidation technique can reduce the power consumption of a data center by up-to 37%.

Unsupervised KPIs-Based Clustering of Jobs in HPC Data Centers.

Performance analysis is an essential task in high-performance computing (HPC) systems, and it is applied for different purposes, such as anomaly detection, optimal resource allocation, and budget planning. HPC monitoring tasks generate a huge number of key performance indicators (KPIs) to supervise the status of the jobs running in these systems. KPIs give data about CPU usage, memory usage, network (interface) traffic, or other sensors that monitor the hardware. Analyzing this data, it is possible to obtain insightful information about running jobs, such as their characteristics, performance, and failures. The main contribution in this paper was to identify which metric/s (KPIs) is/are the most appropriate to identify/classify different types of jobs according to their behavior in the HPC system. With this aim, we had applied different clustering techniques (partition and hierarchical clustering algorithms) using a real dataset from the Galician computation center (CESGA). We concluded that (i) those metrics (KPIs) related to the network (interface) traffic monitoring provided the best cohesion and separation to cluster HPC jobs, and (ii) hierarchical clustering algorithms were the most suitable for this task. Our approach was validated using a different real dataset from the same HPC center.

High Performance Computing (HPC) Data Center for Information as a Service (IaaS) Security Checklist: Cloud Data Governance

High Performance Computing (HPC) Data Center for Information as a Service (IaaS) Security Checklist: Cloud Data Governance

MILP formulations for spatio-temporal thermal-aware scheduling in Cloud and HPC datacenters

This paper focuses on scheduling problems related to the execution of computational jobs in datacenters with thermal constraints. Mixed integer linear programming (MILP) formulations are proposed that encompass both spatial and temporal aspects of the temperature evolution under a unified model. This model takes into account the dynamics of heat production and dissipation in order to schedule jobs at appropriate times on appropriate machines. The proposed MILP formulations are applicable to both high-performance computing (HPC) and Cloud settings, and can target several objectives including energy and makespan minimization, while incorporating the cooling costs and dynamic voltage and frequency scaling capabilities of servers. The applicability and usefulness of our formulations are demonstrated via several HPC and Cloud case-studies.

Reuse of Data Center Waste Heat in Nearby Neighborhoods: A Neural Networks-Based Prediction Model

This paper addresses the problem of data centers’ cost efficiency considering the potential of reusing the generated heat in district heating networks. We started by analyzing the requirements and heat reuse potential of a high performance computing data center and then we had defined a heat reuse model which simulates the thermodynamic processes from the server room. This allows estimating by means of Computational Fluid Dynamics simulations the temperature of the hot air recovered by the heat pumps from the server room allowing them to operate more efficiently. To address the time and space complexity at run-time we have defined a Multi-Layer Perceptron neural network infrastructure to predict the hot air temperature distribution in the server room from the training data generated by means of simulations. For testing purposes, we have modeled a virtual server room having a volume of 48 m3 and two typical 42U racks. The results show that using our model the heat distribution in the server room can be predicted with an error less than 1 °C allowing data centers to accurately estimate in advance the amount of waste heat to be reused and the efficiency of heat pump operation.

Optimization based resource and cooling management for a high performance computing data center

This paper focuses on the problem of reducing energy consumption within high-performance computing data centers, especially for those with a large portion of “small size” jobs. Different from previous works, the efficiency of job scheduling and processing is made as the first priority. To reduce energy from servers while maintaining the processing efficiency of jobs, a new hysteresis computing resource-provisioning algorithm is proposed to adjust the total computing resource reactively. A dynamical thermal model is presented to reflect the relationship between the computational system and cooling system. The proposed model is used to formulate constrained optimal control problems to minimize the energy consumption of the cooling system. Then, a two-step solution is proposed. Firstly, a thermal-aware resource allocation optimizer is developed to decide where the resource should be increased or decreased. Secondly, an economic model predictive controller is designed to adjust the cooling temperature predictively along with the variation of the rack power. Performance of the proposed method is studied through simulations with real job trace. The results show that significant energy saving can be achieved with guaranteed service quality.

Modeling and Simulation of Extreme-Scale Fat-Tree Networks for HPC Systems and Data Centers

As parallel and distributed systems are evolving toward extreme scale, for example, high-performance computing systems involve millions of cores and billion-way parallelism, and high-capacity storage systems require efficient access to petabyte or exabyte of data, many new challenges are posed on designing and deploying next-generation interconnection communication networks in these systems. Fat-tree networks have been widely used in both data centers and high-performance computing (HPC) systems in the past decades and are promising candidates of the next-generation extreme-scale networks. In this article, we present FatTreeSim, a simulation framework that supports modeling and simulation of extreme-scale fat-tree networks with the goal of understanding the design constraints of next-generation HPC and distributed systems and aiding the design and performance optimization of the applications running on these systems. We have systematically experimented FatTreeSim on Emulab and Blue Gene/Q and analyzed the scalability and fidelity of FatTreeSim with various network configurations. On the Blue Gene/Q Mira, FatTreeSim can achieve a peak performance of 305 million events per second using 16,384 cores. Finally, we have applied FatTreeSim to simulate several large-scale Hadoop YARN applications to demonstrate its usability.

Error-Free 56 Gb/s NRZ Modulation of a 1530-nm VCSEL Link

We demonstrate a 1530 nm VCSEL that can operate error-free without DSP or FEC to 56 Gb/s. At 50 Gb/s, error-free operation is attained up to 2 km of SMF. A two-tap FFE driver is used to precompensate the response of the VCSEL. The optical spectrum of the VCSEL under equalization at 50 Gb/s is analyzed and the chirp properties are reported. The low latency of FEC-free NRZ and the distance of 2 km makes these technology suitable for optical links in both high performance computing and large data centers.

Power reduction in HPC data centers: a joint server placement and chassis consolidation approach

Size and number of high-performance data centers are rapidly growing all around the world in recent years. The growth in the leakage power consumption of servers along with its exponential dependence on the ever increasing process variation in nanometer technologies has made it inevitable to move toward variation-aware power reduction strategies in data centers. In this paper, we address the problem of joint server placement and chassis consolidation to minimize power consumption of high-performance computing data centers under process variation. To this end, we introduce two variation-aware server placement heuristics as well as an integer linear programming (ILP)-based server placement method to find the best location of each server in the data center based on its power consumption and the data center heat recirculation model. We then incorporate a novel ILP-based variation-aware chassis consolidation technique to find the optimum task assignment solution under the obtained server placement approach to minimize total power consumption. Experimental results show that by applying the proposed joint variation-aware server placement and chassis consolidation techniques, up to 14.6 % improvement can be obtained at common data center utilization rates compared to state-of-the-art variation-unaware approaches.

Simulation and optimization of HPC job allocation for jointly reducing communication and cooling costs

Performance and energy are critical aspects in high performance computing (HPC) data centers. Highly parallel HPC applications that require multiple nodes usually run for long durations in the range of minutes, hours or days. As the threads of parallel applications communicate with each other intensively, the communication cost of these applications has a significant impact on data center performance. Energy consumption has also become a first-order constraint of HPC data centers. Nearly half of the energy in the computing clusters today is consumed by the cooling infrastructure. Existing job allocation policies either target improving the system performance or reducing the cooling energy cost of the server nodes. How to optimize the system performance while minimizing the cooling energy consumption is still an open question. This paper proposes a job allocation methodology aimed at jointly reducing the communication cost and the cooling energy of HPC data centers. In order to evaluate and validate our optimization algorithm, we implement our joint job allocation methodology in the structural simulation toolkit (SST) – a simulation framework for large-scale data centers. We evaluate our joint optimization algorithm using traces extracted from real-world workloads. Experimental results show that, in comparison to performance-aware job allocation algorithms, our algorithm achieves comparable running times and reduces the cooling power by up to 42.21% across all the jobs.

Monitoring Power Data: A first step towards a unified energy efficiency evaluation toolset for HPC data centers

The energy consumption of High Performance Computing (HPC) systems, which are the key technology for many modern computation-intensive applications, is rapidly increasing in parallel with their performance improvements. This increase leads HPC data centers to focus on three major challenges: the reduction of overall environmental impacts, which is driven by policy makers; the reduction of operating costs, which are increasing due to rising system density and electrical energy costs; and the 20 MW power consumption boundary for Exascale computing systems, which represent the next thousandfold increase in computing capability beyond the currently existing petascale systems. Energy efficiency improvements will play a major part in addressing these challenges.This paper presents a toolset, called Power Data Aggregation Monitor (PowerDAM), which collects and evaluates data from all aspects of the HPC data center (e.g. environmental information, site infrastructure, information technology systems, resource management systems, and applications). The aim of PowerDAM is not to improve the HPC data center's energy efficiency, but is to collect energy relevant data for analysis without which energy efficiency improvements would be non-trivial and incomplete. Thus, PowerDAM represents a first step towards a truly unified energy efficiency evaluation toolset needed for improving the overall energy efficiency of HPC data centers.

The 4 Pillar Framework for energy efficient HPC data centers

Improving energy efficiency has become a major research area not just for commercial data centers but also for high performance computing (HPC) data centers. While many approaches for reducing the energy consumption in data centers and HPC sites have been proposed and implemented, as of today, many research teams focused on improving the energy efficiency of data centers are working independently from others. The main reason being that there is no underlying framework that would allow them to relate their work to achievements made elsewhere. Also, without some frame of correlation, the produced results are either not easily applicable beyond their origin or it is not clear if, when, where, and for whom else they are actually useful. This paper introduces the “4 Pillar Framework for Energy Efficient HPC Data Centers” which can be used by HPC center managers to wholistically evaluate their site, find specific focus areas, classify current research activities, and identify areas for further improvement and research. The 4 pillars are: 1. Building Infrastructure; 2. HPC Hardware; 3. HPC System Software; and 4. HPC Applications. While most HPC centers already implement optimizations within each of the pillars, optimization efforts crossing the pillar boundaries are still rare. The 4 Pillar Framework, however, specifically encourages such cross-pillar optimization efforts. Besides introducing the framework itself, this paper shows its applicability by mapping current research activities in the field of energy efficient HPC conducted at Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities to the framework as reference.

“Cool” Load Balancing for High Performance Computing Data Centers

As we move to exascale machines, both peak power demand and total energy consumption have become prominent challenges. A significant portion of that power and energy consumption is devoted to cooling, which we strive to minimize in this work. We propose a scheme based on a combination of limiting processor temperatures using dynamic voltage and frequency scaling (DVFS) and frequency-aware load balancing that reduces cooling energy consumption and prevents hot spot formation. Our approach is particularly designed for parallel applications, which are typically tightly coupled, and tries to minimize the timing penalty associated with temperature control. This paper describes results from experiments using five different Charm++ and MPI applications with a range of power and utilization profiles. They were run on a 32-node (128-core) cluster with a dedicated air conditioning unit. The scheme is assessed based on three metrics: the ability to control processors' temperature and hence avoid hot spots, minimization of timing penalty, and cooling energy savings. Our results show cooling energy savings of up to 63 percent, with a timing penalty of only 2-23 percent.

Implementing the data center energy productivity metric

As data centers proliferate in size and number, the endeavor to improve their energy efficiency and productivity is becoming increasingly important. We discuss the properties of a number of the proposed metrics of energy efficiency and productivity. In particular, we focus on the Data Center Energy Productivity (DCeP) metric, which is the ratio of useful work produced by the data center to the energy consumed performing that work. We describe our approach for using DCeP as the principal outcome of a designed experiment using a highly instrumented, high-performance computing data center. We found that DCeP was successful in clearly distinguishing different operational states in the data center, thereby validating its utility as a metric for identifying configurations of hardware and software that would improve (or even maximize) energy productivity. We also discuss some of the challenges and benefits associated with implementing the DCeP metric, and we examine the efficacy of the metric in making comparisons within a data center and among data centers.

Rack aware scheduling in HPC data centers: an energy conservation strategy

Energy consumption in high performance computing data centers has become a long standing issue. With rising costs of operating the data center, various techniques need to be employed to reduce the overall energy consumption. Currently, among others there are techniques that guarantee reduced energy consumption by powering on/off the idle nodes. However, most of them do not consider the energy consumed by other components in a rack. Our study addresses this aspect of the data center. We show that we can gain considerable energy savings by reducing the energy consumed by these rack components. In this regard, we propose a scheduling technique that will help schedule jobs with the above mentioned goal. We claim that by our scheduling technique we can reduce the energy consumption considerably without affecting other performance metrics of a job. We implement this technique as an enhancement to the well-known Maui scheduler and present our results. We propose three different algorithms as part of this technique. The algorithms evaluate the various trade-offs that could be possibly made with respect to overall cluster performance. We compare our technique with various currently available Maui scheduler configurations. We simulate a wide variety of workloads from real cluster deployments using the simulation mode of Maui. Our results consistently show about 7 to 14 % savings over the currently available Maui scheduler configurations. We shall also see that our technique can be applied in tandem with most of the existing energy aware scheduling techniques to achieve enhanced energy savings. We also consider the side effects of power losses due to the network switches as a result of deploying our technique. We compare our technique with the existing techniques in terms of the power losses due to these switches based on the results in Sharma and Ranganathan, Lecture Notes in Computer Science, vol. 5550, 2009 and account for the power losses. We there on provide a best fit scheme with the rack considerations. We then propose an enhanced technique that merges the two extremes of node allocation based on rack information. We see that we can provide a way to configure the scheduler based on the kind of workload that it schedules and reduce the effect of job splitting across multiple racks. We further discuss how the enhancement can be utilized to build a learning model which can be used to adaptively adjust the scheduling parameters based on the workload experienced.