MapReduce Scheduling Algorithms Research Articles

Jargon of Hadoop MapReduce scheduling techniques: a scientific categorization

Abstract Recently, valuable knowledge that can be retrieved from a huge volume of datasets (called Big Data) set in motion the development of frameworks to process data based on parallel and distributed computing, including Apache Hadoop, Facebook Corona, and Microsoft Dryad. Apache Hadoop is an open source implementation of Google MapReduce that attracted strong attention from the research community both in academia and industry. Hadoop MapReduce scheduling algorithms play a critical role in the management of large commodity clusters, controlling QoS requirements by supervising users, jobs, and tasks execution. Hadoop MapReduce comprises three schedulers: FIFO, Fair, and Capacity. However, the research community has developed new optimizations to consider advances and dynamic changes in hardware and operating environments. Numerous efforts have been made in the literature to address issues of network congestion, straggling, data locality, heterogeneity, resource under-utilization, and skew mitigation in Hadoop scheduling. Recently, the volume of research published in journals and conferences about Hadoop scheduling has consistently increased, which makes it difficult for researchers to grasp the overall view of research and areas that require further investigation. A scientific literature review has been conducted in this study to assess preceding research contributions to the Apache Hadoop scheduling mechanism. We classify and quantify the main issues addressed in the literature based on their jargon and areas addressed. Moreover, we explain and discuss the various challenges and open issue aspects in Hadoop scheduling optimizations.

MapReduce scheduling algorithms: a review

Recent trends in big data have shown that the amount of data continues to increase at an exponential rate. This trend has inspired many researchers over the past few years to explore new research direction of studies related to multiple areas of big data. The widespread popularity of big data processing platforms using MapReduce framework is the growing demand to further optimize their performance for various purposes. In particular, enhancing resources and jobs scheduling are becoming critical since they fundamentally determine whether the applications can achieve the performance goals in different use cases. Scheduling plays an important role in big data, mainly in reducing the execution time and cost of processing. This paper aims to survey the research undertaken in the field of scheduling in big data platforms. Moreover, this paper analyzed scheduling in MapReduce on two aspects: taxonomy and performance evaluation. The research progress in MapReduce scheduling algorithms is also discussed. The limitations of existing MapReduce scheduling algorithms and exploit future research opportunities are pointed out in the paper for easy identification by researchers. Our study can serve as the benchmark to expert researchers for proposing a novel MapReduce scheduling algorithm. However, for novice researchers, the study can be used as a starting point.

Exact and heuristic MapReduce scheduling algorithms for cloud federation

Geo-distributed big-data processing has recently received much attention since it ensures large-scale and geographically distributed data processing, using Hadoop or Spark, in an efficient, fault-tolerant and reliable manner. The objective of this work is to propose a new geo-distributed MapReduce-based framework and algorithm for federated cloud platforms. A distributed heuristic algorithm, called FDMR (Federated Distributed MapReduce), that takes advantage of data locality, inter-cloud data transfer and high availability of capacities offered by the federation is proposed. The aim of FDMR is to reduce job cost while respecting deadline constraint. The goal of this paper is also to propose an exact MapReduce scheduling model to serve as a baseline for benchmarking and to compare and discuss the heuristic algorithm results. The performance evaluation proves that the proposed algorithm FDMR can improve resource utilization of the cloud federation and consequently reduce cost and job response time while satisfying the deadline constraint.

LEARN ON MAPREDUCE METHOD FOR JOB SCHEDULING BY USING BIGDATA

Many companies are increasingly using Map Reduce for inexperienced massive scale data processing together with personalized marketing , direct mail detection, and brilliant data mining obligations. Cloud computing offer an attractive desire for corporations to hire an appropriate length Hadoop cluster, use resources as a company, and pay simplest for property that has been implemented. One of the open questions in such environments is the amount of property that someone ought to rent from the service provider. Often, a patron goals particular well-known normal performance dreams and the software program desires to entire facts processing by means of the manner of way of a positive time cut-off date. However, currently, the mission of estimating required assets to satisfy software program, overall performance desires is most effective the clients’ obligation. In these , we introduce a unique framework and approach to deal with this problem and to provide a modern beneficial resource sizing and provisioning service in Map Reduce environments. For a Map Reduce technique that desires to be finished in interior a positive time, the challenging profile is constructed from the technique past executions or by way of executing the utility on a smaller statistics set the use of an automatic profiling device. Map Reduce application is used to acquire data in line with the request. To approach massive facts proper scheduling is required to gain extra well-known overall performance. Scheduling is a way of assigning jobs to available assets in a way to decrease hunger and maximize resource usage. Performance of scheduling technique may be advanced with the useful resource of using lessen-off date constraints on jobs. The goal is to examine Map Reduce and notable scheduling algorithms that can be used to benefit better overall performance.

Multiple MapReduce Jobs in Distributed Scheduler for Big Data Applications

The majority of large-scale data intensive applications executed by data centers are based on MapReduce or its open-source implementation, Hadoop. Such applications are executed on large clusters requiring large amounts of energy, making the energy costs a considerable fraction of the data center’s overall costs. Therefore minimizing the energy consumption when executing each MapReduce job is a critical concern for data centers. In this paper, we propose a framework for improving the energy efficiency of MapReduce applications, while satisfying the service level agreement (SLA).We first model the problemof energy-aware scheduling of a single MapReduce job as an Integer Program. We then propose two heuristic algorithms, called energy-aware MapReduce scheduling algorithms (EMRSA-I and EMRSA-II), that find the assignments of map and reduce tasks to the machine slots in order to minimize the energy consumed when executing the application. Our algorithm able to find near optimal job schedules consuming approximately 40 percent less energy on average than the schedules obtained by a common practice scheduler that minimizes the makespan.

Energy-Aware Scheduling of MapReduce Jobs for Big Data Applications

The majority of large-scale data intensive applications executed by data centers are based on MapReduce or its open-source implementation, Hadoop. Such applications are executed on large clusters requiring large amounts of energy, making the energy costs a considerable fraction of the data center’s overall costs. Therefore minimizing the energy consumption when executing each MapReduce job is a critical concern for data centers. In this paper, we propose a framework for improving the energy efficiency of MapReduce applications, while satisfying the service level agreement (SLA). We first model the problem of energy-aware scheduling of a single MapReduce job as an Integer Program. We then propose two heuristic algorithms, called energy-aware MapReduce scheduling algorithms (EMRSA-I and EMRSA-II), that find the assignments of map and reduce tasks to the machine slots in order to minimize the energy consumed when executing the application. We perform extensive experiments on a Hadoop cluster to determine the energy consumption and execution time for several workloads from the HiBench benchmark suite including TeraSort, PageRank, and K-means clustering, and then use this data in an extensive simulation study to analyze the performance of the proposed algorithms. The results show that EMRSA-I and EMRSA-II are able to find near optimal job schedules consuming approximately 40 percent less energy on average than the schedules obtained by a common practice scheduler that minimizes the makespan.

Evaluating map reduce tasks scheduling algorithms over cloud computing infrastructure

SummaryEfficiently scheduling MapReduce tasks is considered as one of the major challenges that face MapReduce frameworks. Many algorithms were introduced to tackle this issue. Most of these algorithms are focusing on the data locality property for tasks scheduling. The data locality may cause less physical resources utilization in non‐virtualized clusters and more power consumption. Virtualized clusters provide a viable solution to support both data locality and better cluster resources utilization. In this paper, we evaluate the major MapReduce scheduling algorithms such as FIFO, Matchmaking, Delay, and multithreading locality (MTL) on virtualized infrastructure. Two major factors are used to test the evaluated algorithms: the simulation time and the energy consumption. The evaluated schedulers are compared, and the results show the superiority and the preference of the MTL scheduler over the other existing schedulers. Also, we present a comparison study between virtualized and non‐virtualized clusters for MapReduce tasks scheduling. Copyright © 2015 John Wiley & Sons, Ltd.

Classification Framework of MapReduce Scheduling Algorithms

A MapReduce scheduling algorithm plays a critical role in managing large clusters of hardware nodes and meeting multiple quality requirements by controlling the order and distribution of users, jobs, and tasks execution. A comprehensive and structured survey of the scheduling algorithms proposed so far is presented here using a novel multidimensional classification framework. These dimensions are (i) meeting quality requirements, (ii) scheduling entities, and (iii) adapting to dynamic environments; each dimension has its own taxonomy. An empirical evaluation framework for these algorithms is recommended. This survey identifies various open issues and directions for future research.

Survey on MapReduce Scheduling Algorithms

MapReduce is a programming model used by Google to process large amount of data in a distributed computing environment. It is usually used to perform distributed computing on clusters of computers. Computational processing of data stored on either a file system or a database usually occurs. MapReduce takes the advantage of locality of data, processing data on or near the storage areas, thereby avoiding unnecessary data transmission. The simplicity of the programming model and the automatic handling of node failures hiding the complexity of fault tolerance make MapReduce to be used for both commercial and scientific applications. As MapReduce clusters have become popular these days, their scheduling is one of the important factor which is to be considered. In order to achieve good performance a MapReduce scheduler must avoid unnecessary data transmission. Hence different scheduling algorithms for MapReduce are necessary to provide good performance. This paper provides an overview of four different scheduling algorithms for MapReduce namely; Scheduling algorithm in Hadoop, Longest Approximate Time to End (LATE) MapReduce scheduling algorithm, Self-Adaptive MapReduce(SAMR) scheduling algorithm and Enhanced Self-Adaptive MapReduce scheduling algorithm(ESAMR). An overview of these techniques is provided through this paper. Advantages and disadvantages of these algorithms are identified.

A Comprehensive View of Hadoop MapReduce Scheduling Algorithms

A Comprehensive View of Hadoop MapReduce Scheduling Algorithms

Improving MapReduce Scheduling Algorithm Using Prediction-Based Application Model in the Cloud

Improving MapReduce Scheduling Algorithm Using Prediction-Based Application Model in the Cloud