Multi-queue scheduling of heterogeneous jobs in hybrid geo-distributed cloud environment

Abstract

In hybrid geo-distributed clouds, there is a technique named cloud bursting in which applications are handled in the private cloud with less expenses and burst into public clouds when the resources of the private cloud run out. However, how to deploy heterogeneous jobs in heterogeneous hybrid cloud environment is still a challenge. In this paper, a multi-queue scheduling approach of heterogeneous jobs for cloud bursting is proposed. In the private cloud, jobs are classified into I/O-intensive and CPU-intensive jobs, and nodes are divided into main I/O and CPU resource pools. Jobs are dispatched to corresponding resource pools to reduce the job execution time in heterogeneous cloud environment. A genetic algorithm is applied to schedule jobs to optimal job queues, which can reduce the job waiting time. Then, the execution time of each task is predicted by BP neural network. Jobs with high priority will be allocated to resources with the earliest finish time in the private cloud according to the prediction results. If the private cloud cannot meet the demand of users, public clouds with minimal costs will be applied. Experiments show that our proposed algorithm can reduce the average job response time and improve the throughput of the private cloud. It also can reduce the average task waiting time, average task execution time and average task response time significantly. Moreover, the costs of the hybrid clouds are reduced.