Towards Efficient Mapping, Scheduling, and Execution of HPC Applications on Platforms in Cloud

Abstract

The advantages of pay-as-you-go model, elasticity, and the flexibility and customization offered by virtualization make cloud computing an attractive option for meeting the needs of some High Performance Computing (HPC) users. However, there is a mismatch between cloud environments and HPC requirements. The poor interconnect and I/O performance in cloud, HPC-agnostic cloud schedulers, and the inherent heterogeneity and multi-tenancy in cloud are some bottlenecks for effective HPC in cloud. Our primary thesis is that cloud is suitable for some HPC applications not all applications, and for those applications, cloud can be more cost-effective compared to typical dedicated HPC platforms using intelligent application-to-platform mapping, HPC-aware cloud schedulers, and cloud-aware HPC execution and parallel runtime system. To address the challenges, and to exploit the opportunities offered by HPC-clouds, we make Open-Stack Nova scheduler HPC-aware and Charm++ parallel runtime system cloud-aware. We demonstrate that our techniques resulting significant improvement in cost (up to 60%), performance (up to 45%), and throughput (up to 32%) for HPC in cloud; helping cloud users gain confidence in the capabilities of cloud for HPC, and cloud providers run a more profitable business.