A Fine-Grained and Dynamic MapReduce Task Scheduling Scheme for the Heterogeneous Cloud Environment

Abstract

MapReduce framework is becoming more and more popular in various applications. However, Hadoop is a seriously limited by its MapReduce scheduler which does not work well in the heterogeneous environment. LATE MapReduce scheduling algorithm takes heterogeneous environment into consideration. However, it falls short of solving the poor performance due to the static manner during computing the tasks progress. In order to improve the cluster performance in a heterogeneous cloud environment, FiGMR -- a Fine-Grained and dynamic MapReeduce scheduling algorithm, is proposed. FiGMR can significantly reduce the tasks execution time and improve the resources utilization. FiGMR includes historical and real-time online information obtained from each node to select the appropriate parameters to find the real slow task dynamically. Meanwhile, in order to further improve the cluster performance, FiGMR classifies map nodes into high-performance map node and low-performance map node. FiGMR classifies slow tasks into slow map tasks and slow reduce tasks. Map/Reduce slow nodes means nodes which execute map/reduce tasks using a longer time than most other nodes. In this way, FiGMR launches backup map tasks on nodes which are high-performance map nodes.