Impacts of Virtualization Technologies on Hadoop

Abstract

MapReduce is an emerging program model in large-scale parallel applications while virtual machine environments are basic computing units in cloud computing. With advantages of these two rising technologies, we can build up a convenient parallel programming environment and make effective use of system resources in cloud computing. In this paper, we describe the effects of different virtualization technologies which one of MapReduce running environments, Hadoop, works in. Then we compared the Hadoop Distributed File System performance running in three types of virtual machines, analyze the resource utility of MapReduce using virtual machines monitor tools which detect the status of task trackers and test MapReduce performance in the cases of different numbers of virtual machines with the same total memories. Experiment results show that Xen performs better than other virtual machines both on performance and stability. And the tools of virtual machine monitor report information of MapReduce different stages and resources utility can greatly help us design a adaptive scheduling. Moreover, better performance can be achieved with more virtual machines under the adequate memory of every virtual machines.