An efficient MapReduce scheduling scheme for processing large multimedia data

Abstract

In this paper, we propose a scheduling scheme to minimize the deadline miss of jobs to which deadlines are assigned when processing large multimedia data such as video and image in MapReduce frameworks. The proposed scheme checks the satisfaction of data locality to process assigned jobs within a time limit and considers whether I/O load and deadline requirement are satisfied. If jobs are run in a node with excessive I/O load, multimedia data from the replica node can be utilized to improve a job task processing speed. If available nodes are not found due to expected job completion time exceeding the deadline, the job tasks in nodes whose deadlines are available are paused temporarily to shorten the job completion time. In addition, speculative tasks and hot data block replication are employed to prevent the overall deadline miss ratio from increasing due to the repetition of job pauses whose deadlines are available for the purpose of processing urgent jobs quickly. The speculative task is a technique for assigning the same job to other nodes redundantly and for taking the result from the node that completes the job first and then cancelling the other jobs assigned previously. To verify the superiority of the proposed scheme, a performance evaluation is conducted by comparing it with the existing scheme. The performance evaluation result showed that the proposed scheme reduced completion time by 13.8 % and improved the deadline success ratio by 11 % compared with those of the existing scheme on average.