Improvement of Makespan and TCTime in Dynamic Job Ordering and Slot Utilization for MapReduce Workloads

Abstract

The amount of data generated in today's environment is increasing at an exponential rate. These data are structured, semi-structured, or unstructured in some cases. Processing vast amounts of data is a difficult task. MapReduce is a technique for processing large amounts of data. For storing big data sets, Hadoop data file system (HDFS) is employed. A MapReduce workload is made up of a number of jobs, each of which has multiple map tasks and numerous reduce tasks. In this paper, we propose the Shortest Task Ordering (SJA) algorithm for optimising Mkspan (Makespan) and TCTime (Total completion time) for MapReduce Workloads using dynamic job ordering and slot design. We conducted a comparison analysis on data sets of various sizes. By comparing objective measurements such as Mkspan and TCTime, the experimental results show that performance in terms of speed has improved. Each set of jobs, such as WordCount, CharCount, LineCount, and Anagram, displays comparative improvement in our work. When compared to previous algorithms such as MkJR and MkTctJR, the results demonstrate an improvement in time efficiency, slot usage, and execution speed. In terms of Mkspan and TCTime, the Shortest Task Assigned (SJA) method for job ordering yielded results that were up to 95% better than MkJR. In comparison to the previous algorithms MkSfJR and MkTctSfJR, there is also an increase in slot usage. In terms of Mkspan and TCTime, the Shortest Job Assigned (SJA) algorithm achieved results that were up to 150 percent better than MkSfJR.