Design and Implementation of Virtual Hadoop Cluster on Private Cloud

Abstract

Virtualization has made it feasible to deploy hadoop in cloud environment. Virtualized hadoop offers unique benefits like setting up a cluster in short time, flexibility to use variety of hardware (SAN, NAS, DAS), high availability and many more. With companies like Google, Microsoft, Rackspace and IBM providing their own infrastructure for cloud service more and more business is assumed to move on cloud in the near future. Apart from public cloud business can make use of private, community or hybrid cloud deployment model. In this paper, the focus is on private cloud deployment which offers its own benefits like security, reduced cost, more control over resources etc. The design and implementation of private cloud using Xen 6.5 bare metal hypervisor is discussed in this paper. Further it also discusses deploying hadoop as service on the cloud with the help of shell script. For experimental purpose 8 physical hosts are connected to 60 Tb SAN with QLogic 20-Port 8 Gb SAN switch module which provides fiber connectivity to the storage. Finally, the performance of hadoop on the cloud is evaluated.