Virtual Cluster Deployment Model for Large-Scale Data Processing Jobs

Abstract

The virtual cluster needs to deploy virtual nodes with different computing modes for Large-scale data computing jobs including many tasks with different computing modes. There are resource unbalance among the virtual cluster subsets, which is consisted of virtual nodes with the same computing mode. So the resource utilization of virtual cluster is poor due to the time-varying and complexity of workloads. To solve these issues, a novel dynamic deployment model based on Docker was proposed to optimize resource configuration of virtual cluster. Hybrid deployment scheme includes coarse-grained deployment and fine-grained deployment in order to absorb the mutation of resource demand when processing the jobs encapsulated with complex computing logic. This deployment model can reduce job waiting time and improve performance by providing more resource for running jobs. The coarse-grained deployment mechanism has the ability of decreasing deploying overhead and improving stability from global optimal perspective. And the fine-grained deployment mechanism can increases the deploying accuracy from local optimal perspective. The experiments show that this model can improve the execution efficiency of job by 2.3 percent compared with static deploying scheme and improve the virtual CPU utilization about 7.9 to 25.9 percent compared with other approaches.