Distributed file load rebalancing methodology for map reduce system

Abstract

Map reduce systems are widely used to solve large computing work. Employing a map reduce system in distributed file system in which there are master node and worker nodes. This master node splits the jobs in to smaller units and submits those jobs to the worker nodes. The worker nodes process the spitted work and send the results to the master node. The master node collects all the results from the worker nodes and consolidates the results and gives response to the client. The nodes may tend to fail in a networked systems. If a node is failed then the jobs running in the node will be moved to other node. The load will be balanced among the worker nodes. The master node takes care of doing this balancing. But there is problem in which load balancing a node comes under heavy load and not able to compute jobs faster. This project addresses the above problem in which the worker nodes itself will have a distributed load rebalancing algorithm among themselves instead of the master node allocates the same. This kind of approach will tends to work faster than the master node reallocates the unit of work.