DCUDA

Abstract

In clouds and data centers, GPU servers which consist of multiple GPUs are widely deployed. Current state-of-the-art GPU scheduling algorithm are static in assigning to different GPUs. These algorithms usually ignore the dynamics of the GPU utilization and are often inaccurate in estimating resource demand before assigning/running applications, so there is a large opportunity to further load balance and to improve GPU utilization. Based on CUDA (Compute Unified Device Architecture), we develop a runtime system called DCUDA which supports dynamic scheduling of between multiple GPUs. In particular, DCUDA provides a realtime and lightweight method to accurately monitor the resource demand of and GPU utilization. Furthermore, it provides a universal migration facility to migrate running applications between GPUs with negligible overhead. More importantly, DCUDA transparently supports all CUDA without changing their source codes. Experiments with our prototype system show that DCUDA can reduce 78.3% of overloaded time of GPUs on average. As a result, for different workloads consisting of a wide range we studied, DCUDA can reduce the average execution time of by up to 42.1%. Furthermore, DCUDA also reduces 13.3% energy in the light load scenario.