Skyway: A Seamless Solution for Bursting Workloads from On-Premises HPC Clusters to Commercial Clouds

Abstract

Interest in cloud computing within the High Performance Computing (HPC) community continues to grow, but barriers to wide-spread adoption remain over concerns of cost, support of low-latency network applications, and data management. Despite these barriers, tremendous potential still exists for cloud computing to enhance HPC through the high scalability and elasticity it provides. HPC workloads and hardware technologies continue to rapidly evolve and become increasingly diversified, which poses a challenge for on-premises resources in production since they are not easily adaptable or immediately expandable. A hybrid infrastructure, with the ability of “burst-to-cloud”, can combine advantages from the on-premises and cloud resource spaces to overcome the challenges and barriers present with each. The Research Computing Center (RCC) at The University of Chicago has developed such a solution, named Skyway, that incorporates multi-cloud computing resources as elastic extensions of its on-premises HPC infrastructure. A system-level software package is also developed to interface with the Simple Linux Utility for Resource Management (SLURM) scheduler and cloud SDKs, that can burst HPC workloads to the cloud, providing users a seamless experience when interacting with both on-premises and cloud systems. Skyway offers the flexibility in interfacing with one or more cloud providers, with current use cases utilizing both Amazon AWS and Google GCP via Skyway. Other cloud providers, such as Microsoft Azure and Oracle, can also be utilized through Skyway, which we plan to explore further in the near future.