An adaptive data transfer algorithm using block device reconfiguration in virtual MapReduce clusters

Abstract

With the proliferation of cloud computing and virtual machine technologies, MapReduce applications are increasingly deployed in clouds to leverage the full potential of cloud computing environments. However, the MapReduce, which is generally used for processing large amount of data, suffers from the I/O virtualization overheads and resource competitions among virtual machines when it is run on virtual clouds. This paper proposes an adaptive data transfer algorithm in virtual MapReduce clusters. The proposed algorithm utilizes a block device reconfiguration scheme, where a block device attached to a virtual machine can be dynamically detached and reattached to other virtual machines hosted in the same physical machine. By reconfiguring the block devices, we can easily move files across different virtual machines located at the same physical machine without any network transfers between virtual machines. When the output of each map task is transferred to the reducer, this algorithm adaptively determines an appropriate transfer method between network transfer and block device reconfiguration based on current CPU utilization values and the data size for the transfer. Even in the case of data transfer between virtual machines across multiple physical machines, we can remove the transfer overheads between the virtual machine and the driver domain, which results in reducing the data transfer time and performance effects to other virtual machines in the shuffle phase. We have implemented our algorithm in Hadoop MapReduce. The benchmarking results show that the overheads incurred by transferring data from mapper virtual machines to reducer virtual machines are minimized and the execution times of MapReduce applications are shortened.