Efficient High Performance Computing in the Cloud

Abstract

The advantages of pay-as-you-go model, elasticity, and the flexibility and customization offered by virtualization make cloud computing an attractive economical option for meeting the needs of some HPC users. However, there is a mismatch between current cloud environments and HPC requirements. HPC is performance-oriented, whereas clouds are cost and resource-utilization oriented. The poor interconnect and I/O performance in cloud, HPC-agnostic cloud schedulers, and the inherent heterogeneity and multi-tenancy in cloud are some bottlenecks for HPC in cloud. This means that the tremendous potential of cloud for both HPC users and providers remains underutilized. In this talk, we will go beyond the common research question: what is the performance of HPC in cloud? and present our research on how can we perform cost-effective and efficient HPC in cloud? To this end, we will present the complementary approach of making clouds HPC-aware, and HPC runtime system cloud-aware. Through comprehensive HPC performance and cost analysis, HPC-aware VM placement, interference-aware VM consolidation, Multi-dimensional Online Bin Packing, malleable jobs, and cloud-aware HPC load balancing, we demonstrate significant benefits for both: users and cloud providers in terms of cost (up to 60%), performance (up to 45%), and throughput (up to 32%).