Virtual cluster optimisation for MapReduce-like applications

Abstract

Infrastructure-as-a-service clouds are becoming ubiquitous for provisioning virtual machines on demand. Cloud service providers expect to use the least resources to deliver the best services. As users frequently request virtual machines to build virtual clusters and run MapReduce-like jobs for big data processing, cloud service providers intend to optimise the virtual cluster to minimise network latency and subsequently reduce data movement cost. In this paper, we focus on the virtual machine placement issue for provisioning virtual clusters with minimum network latency in clouds. We define the distance as the latency between virtual machines and use it to measure the affinity of a virtual cluster. Such metric of distance indicates the considerations of virtual machine placement and the topology of physical nodes in clouds. Then, we formulate our problem as the classical shortest distance problem and solve it by building an integer programming model. A greedy virtual machine placement algorithm is designed to get a compact virtual cluster. Furthermore, an improved heuristic algorithm is also presented for achieving a global resource optimisation. The simulation results verify our algorithms and the experiment results validate the improvement achieved by our approaches.