Dynamic Scheduling for Speculative Execution to Improve MapReduce Performance in Heterogeneous Environment

Abstract

MapReduce framework allows users to quickly develop big-data applications and process big-data effectively. However, unexpected malfunction may be found in cloud environment because a distributed system consists of several hardware, and this malfunction often causes delay of overall processing. MapReduce framework provides Speculative Execution (SE). SE reduces delay in a homogeneous environment by assigning delayed tasks to additional nodes. As cloud computing prevails, cloud computing environment is moving from homogeneous to heterogeneous. Original SE is not perfect and sometimes produces inefficient result in a heterogeneous environment. This paper proposes Dynamic Scheduling for Speculative Execution (DSSE) which enhances performance in a heterogeneous environment by improving existing SE. DSSE prevents wasted SE since it calculates processing capability of each node more objectively and precisely. DSSE has reduced entire processing time approximately 10% compared to original SE. Success rate of SE was 100%.