Reduce Task Scheduling Research Articles

TMaR: a two-stage MapReduce scheduler for heterogeneous environments

In the context of MapReduce task scheduling, many algorithms mainly focus on the scheduling of Reduce tasks with the assumption that scheduling of Map tasks is already done. However, in the cloud deployments of MapReduce, the input data is located on remote storage which indicates the importance of the scheduling of Map tasks as well. In this paper, we propose a two-stage Map and Reduce task scheduler for heterogeneous environments, called TMaR. TMaR schedules Map and Reduce tasks on the servers that minimize the task finish time in each stage, respectively. We employ a dynamic partition binder for Reduce tasks in the Reduce stage to lighten the shuffling traffic. Indeed, TMaR minimizes the makespan of a batch of tasks in heterogeneous environments while considering the network traffic. The simulation results demonstrate that TMaR outperforms Hadoop-stock and Hadoop-A in terms of makespan and network traffic and achieves by an average of 29%, 36%, and 14% performance using Wordcount, Sort, and Grep benchmarks. Besides, the power reduction of TMaR is up to 12%.

An Optimal Locality-Aware Task Scheduling Algorithm Based on Bipartite Graph Modelling for Spark Applications

In the distributed computing framework of Spark, cross-node/rack data transfer produced by map tasks and reduce tasks are common problems resulting in performance degradation, such as prolonging of entire execution time and network congestion. To address these problems, this article utilizes the bipartite graph modelling to propose an optimal locality-aware task scheduling algorithm. By considering global optimality, the algorithm can generate the optimal scheduling solution for both the map tasks and the reduce tasks for data locality. Because of the different communication modes, this article uses a unified graph to model the map task scheduling and the reduce task scheduling respectively. Then, by calculating the communication cost matrix of tasks, we formulate an optimal task scheduling scheme to minimize overall communication cost and transform the problem as the well-known graph problem: minimum weighted bipartite matching (MWBM), which can be resolved by Kuhn-Munkres algorithm. In addition, this article proposes a locality-aware executor allocation strategy to improve the data locality further. We implement our algorithm and strategy in Spark-2.4.1 and evaluate its performance using several representative micro-benchmarks, macro-benchmarks, and HiBench benchmark suite. The experimental results verify that by reducing the network traffic and access latency, the proposed algorithm can improve the job performance substantially compared to some other task scheduling algorithms.

Data-local Reduce Task Scheduling

Inspired by the victory of Apache's Hadoop this paper suggests a new reduce task scheduler. Hadoop is an open source implementation of Google's MapReduce framework. Programs which are written in this functional style are automatically executed and parallelized on a large cluster of commodity machines. The details how to partition the input data, setting up the program's for execution across a set of machines, handling failures of machine and managing the necessary inter-device communication is taken care by runtime system. In the current versions of Hadoop, the map tasks are scheduled with respect to the locality of their inputs in order to shrink network traffic and improve performance. On the other hand, the reduce tasks are scheduled without taking into consideration data locality leading to ruin the performance at requesting nodes. In this paper, we use data locality that is natural with reduce tasks. To accomplish the same, we schedule them on nodes that will result in least amount data- local traffic. Experimental results signify an 11-80 percent decrease in the number of bytes shuffled in a Hadoop cluster.

Improving map reduce task scheduling and micro-partitioning mechanism for mobile cloud multimedia services

Over the several decades, there is a massive improvement in the computer technology which leads to infinite number of resources in all over the world. Many computing devices have to generate data that comes from various domains. In order to reduce the time complexity and the storage space of data, a novel technique namely map reduce programming model has been proposed to divide the workload among computers in a network to enhance the performance. To rectify the challenging issue of uneven data distribution, and also to enhance the process of load balancing along with memory consumption of computer, data sampling is highly preferred. To enhance the accuracy in scheduling, an innovative method called map reduce task scheduling algorithm is proposed for job deadline constraints. This algorithm classifies the nodes into several levels in heterogeneous clusters. Under this algorithm, a novel data distribution model has been elucidated in which it distributes data according to the node's capacity level respectively.

A Reduce Task Scheduler for MapReduce with Minimum Transmission Cost Based on Sampling Evaluation

MapReduce is a popular framework for processing large datasets in parallel over a cluster. It has gained wide attention for its high scalability, reliability and low cost. However, its performance may be degraded by excessive network traffic when processing jobs, for such two problems as data locality in reduce task scheduling and partitioning skew. We propose a Minimum Transmission Cost Reduce task Scheduler (MTCRS) based on sampling evaluation to solve the two problems. The MTCRS takes the waiting time of each reduce task and the transmission cost set as indicators to decide appropriate launching locations for Reduce tasks. The transmission cost set is computed by a mathematical model, in which the parameters are the sizes and the locations of intermediate data partitions generated by Average Reservoir Sampling (ARS) algorithm. The experiments show that the MTCRS reduces network traffic by 8.4% compared with Fair scheduler.

An Optimized Algorithm for Reduce Task Scheduling

In this paper, we propose a novel algorithm to solve the starving problem of the small jobs and reduce the process time of the small jobs on Hadoop platform. Current schedulers of MapReduce/Hadoop are quite successful in achieving data locality and scheduling the reduce tasks with a greedy algorithm. Some jobs may have hundreds of map tasks and just several reduce tasks, in which case, the reduce tasks of the large jobs require more time for waiting, which will result in the starving problem of the small jobs. Since the map tasks and the reduce tasks are scheduled separately, we can change the way the scheduler launches the reduce tasks without affecting the map phase. Therefore we develop an optimized algorithm to schedule the reduce tasks with the shortest remaining time (SRT) of the map tasks. We apply our algorithm to the fair scheduler and the capacity scheduler, which are both widely used in real production environment. The evaluation results show that the SRT algorithm can decrease the process time of the small jobs effectively.

Investigating MapReduce framework extensions for efficient processing of geographically scattered datasets

In this paper, we investigate real-world scenarios in which MapReduce programming model and specifically Hadoop framework could be used for processing large-scale, geographically scattered datasets. We propose an Adaptive Reduce Task Scheduling (ARTS) algorithm and evaluate it on a distributed Hadoop cluster involving multiple datacenters as well as the on a shared Hadoop cluster. The evaluation demonstrates that the ARTS algorithm outperforms the default Reduce phase scheduling algorithm in Hadoop framework.

A Data Locality and Skew Aware Task Scheduler for MapReduce in Cloud Computing

Inspired by the success and the increasing prevalence of MapReduce, this work proposes a novel MapReduce task scheduler. MapReduce is by far one of the most successful realizations of large-scale, data-intensive, cloud computing platforms. As compared to traditional programming models, MapReduce automatically and efficiently parallelizes computation by running multiple Map and/or Reduce tasks over distributed data across multiple machines. Hadoop, an open source implementation of MapReduce, schedules Map tasks in the vicinity of their input splits seeking diminished network traffic. However, when Hadoop schedules Reduce tasks, it neither exploits data locality nor addresses data partitioning skew inherent in many MapReduce applications. Consequently, MapReduce experiences a performance penalty and network congestion as observed in our experimental results. Recently there has been some work concerned with leveraging data locality in Reduce task scheduling. For instance, one study suggests a locality-aware mechanism that inspects Map inputs and predicts corresponding consuming reducers. The input splits are subsequently assigned to Map tasks near the future reducers. While such a scheme addresses the problem, it targets mainly public-resource grids and doesn't fully substantiate the accuracy of the suggested prediction process. In this study, we propose Locality-Aware Skew-Aware Reduce Task Scheduler (LASAR), a practical strategy for improving MapReduce performance in clouds. LASAR attempts to schedule each reducer at its center-of-gravity node. It controllably avoids scheduling skew, a situation where some nodes receive more reducers than others, and promotes effective pseudo-asynchronous Map and Reduce phases resulting in earlier completion of submitted jobs, diminished network traffic, and better cluster utilization. vWe implemented LASAR in Hadoop-0.20.2 and conducted extensive experimentations to evaluate its potential. We found that it outperforms current Hadoop by 11%, and by up to 26% for the utilized benchmarks. We believe LASAR is applicable to several cloud computing environments and multiple essential applications, including but not limited to shared environments and scientific applications. In fact, a large body of work observed partitioning skew in many of critical scientific applications. LASAR paves the way for these applications, and others, to get effectively ported to various clouds.