Locality-aware Task Scheduling Research Articles

Prediction-Based and Locality-Aware Task Scheduling for Parallelizing Video Transcoding Over Heterogeneous MapReduce Cluster

MapReduce is a popular programming model in cloud computing to deal with the high computational task, such as video transcoding. It splits the video (task) into multiple segments (subtasks) and transcodes them in parallel in cluster. Due to the complexity of video transcoding and the poor performance of heterogeneous MapReduce cluster, scheduling these subtasks to minimize the total transcoding time is still a challenge. In this paper, we propose a prediction-based and locality-aware task scheduling (PLTS) method for parallelizing video transcoding over heterogeneous MapReduce cluster. First, we analyze video decoding and encoding technologies and predict the segment transcoding complexity, which can provide a foundational base for the following scheduling. Second, we attempt to schedule subtasks on machines that contain the related input data, which are referred to as data locality, so as to reduce large-scale data movement and data transfer during the mapping phase. Third, we formulate the scheduling as a job shop scheduling problem and propose a heuristic PLTS algorithm. It combines the benefits of two traditional heuristic scheduling algorithms, Max–Min and Min–Min, to make load balancing in cluster and short the total transcoding time. The experimental results also show the efficiency of our algorithm.

Energy-efficient hadoop for big data analytics and computing: A systematic review and research insights

As the demands for big data analytics keep growing rapidly in scientific applications and online services, MapReduce and its open-source implementation Hadoop gained popularity in both academia and enterprises. Hadoop provides a highly feasible solution for building big data analytics platforms. However, defects of Hadoop are also exposed in many aspects including data management, resource management, scheduling policies, etc. These issues usually cause high energy consumption when running MapReduce jobs in Hadoop clusters. In this paper, we review the studies on improving energy efficiency of Hadoop clusters and summarize them in five categories including the energy-aware cluster node management, energy-aware data management, energy-aware resource allocation, energy-aware task scheduling and other energy-saving schemes. For each category, we briefly illustrate its rationale and comparatively analyze the relevant works regarding their advantages and limitations. Moreover, we present our insights and figure out possible research directions including energy-efficient cluster partitioning, data-oriented resource classification and provisioning, resource provisioning based on optimal utilization, EE and locality aware task scheduling, optimizing job profiling with machine learning, elastic power-saving Hadoop with containerization and efficient big data analytics on Hadoop. On one hand, the summary of studies on energy-efficient Hadoop presented in this paper provides useful guidance for the developers and users to better utilize Hadoop. On the other hand, the insights and research trends discussed in this work may inspire the relevant research on improving the energy efficiency of Hadoop in big data analytics.

Locality-aware task scheduling for homogeneous parallel computing systems

In systems with complex many-core cache hierarchy, exploiting data locality can significantly reduce execution time and energy consumption of parallel applications. Locality can be exploited at various hardware and software layers. For instance, by implementing private and shared caches in a multi-level fashion, recent hardware designs are already optimised for locality. However, this would all be useless if the software scheduling does not cast the execution in a manner that promotes locality available in the programs themselves. Since programs for parallel systems consist of tasks executed simultaneously, task scheduling becomes crucial for the performance in multi-level cache architectures. This paper presents a heuristic algorithm for homogeneous multi-core systems called locality-aware task scheduling (LeTS). The LeTS heuristic is a work-conserving algorithm that takes into account both locality and load balancing in order to reduce the execution time of target applications. The working principle of LeTS is based on two distinctive phases, namely; working task group formation phase (WTG-FP) and working task group ordering phase (WTG-OP). The WTG-FP forms groups of tasks in order to capture data reuse across tasks while the WTG-OP determines an optimal order of execution for task groups that minimizes the reuse distance of shared data between tasks. We have performed experiments using randomly generated task graphs by varying three major performance parameters, namely: (1) communication to computation ratio (CCR) between 0.1 and 1.0, (2) application size, i.e., task graphs comprising of 50-, 100-, and 300-tasks per graph, and (3) number of cores with 2-, 4-, 8-, and 16-cores execution scenarios. We have also performed experiments using selected real-world applications. The LeTS heuristic reduces overall execution time of applications by exploiting inter-task data locality. Results show that LeTS outperforms state-of-the-art algorithms in amortizing inter-task communication cost.

Locality-Aware Task Scheduling and Data Distribution for OpenMP Programs on NUMA Systems and Manycore Processors

Performance degradation due to nonuniform data access latencies has worsened on NUMA systems and can now be felt on-chip in manycore processors. Distributing data across NUMA nodes and manycore processor caches is necessary to reduce the impact of nonuniform latencies. However, techniques for distributing data are error-prone and fragile and require low-level architectural knowledge. Existing task scheduling policies favor quick load-balancing at the expense of locality and ignore NUMA node/manycore cache access latencies while scheduling. Locality-aware scheduling, in conjunction with or as a replacement for existing scheduling, is necessary to minimize NUMA effects and sustain performance. We present a data distribution and locality-aware scheduling technique for task-based OpenMP programs executing on NUMA systems and manycore processors. Our technique relieves the programmer from thinking of NUMA system/manycore processor architecture details by delegating data distribution to the runtime system and uses task data dependence information to guide the scheduling of OpenMP tasks to reduce data stall times. We demonstrate our technique on a four-socket AMD Opteron machine with eight NUMA nodes and on the TILEPro64 processor and identify that data distribution and locality-aware task scheduling improve performance up to 69% for scientific benchmarks compared to default policies and yet provide an architecture-oblivious approach for programmers.

Resource Allocation and Scheduling in Cloud Computing: Policy and Algorithm

ABSTRACTCloud computing is a new distributed commercial computing model that aims at providing computational resources or services to users over a network in a low-cost manner. Resource allocation and scheduling (RAS) is the key focus of cloud computing, and its policy and algorithm have a direct effect on cloud performance and cost. This paper presents five major topics in cloud computing, namely locality-aware task scheduling; reliability-aware scheduling; energy-aware RAS; Software as a Service (SaaS) layer RAS; and workflow scheduling. These five topics are then classified into three parts: performance-based RAS; cost-based RAS; and performance- and cost-based RAS. A number of existing RAS policies and algorithms are discussed in detail accordingly with regard to their given parameters. In addition, a comparative analysis of five identified problems with their representative algorithms is performed. Finally, some future research directions of cloud RAS are pointed out.