Multi-job Scheduling Research Articles

Multi-Job Scheduling and Optimization Model for Cloud Computing

Multi-Job Scheduling and Optimization Model for Cloud Computing

Online job scheduling for distributed machine learning in optical circuit switch networks

Networking has become a well-known performance bottleneck for distributed machine learning (DML). Although lots of works have focused on accelerating the communication process of DML, they ignore the impact of the physical network on the DML performance. Concurrently, optical circuit switches (OCSes) are increasingly applied in data centers and clusters, which can fundamentally improve DML performance. It is worth noting that the non-negligible OCS reconfiguration delay makes OCS scheduling algorithms have a great impact on the upper application performance. However, existing OCS scheduling solutions are not suitable for DML jobs due to the iterative nature of DML jobs and their interleaving characteristics of communication and computation stages. Therefore, in this paper, we study the online multi-job scheduling for DML in OCS networks. Firstly, we propose heaviest-load-first (HLF), a heuristic algorithm for intra-job scheduling, which is based on the fact that the completion time of flows on the heaviest load port has a significant impact on the job completion time. Furthermore, we present Shortest Weighted Remaining Time First (SWRTF) algorithm for inter-job scheduling. In SWRTF, an available DML job is scheduled when the served job moves from communication stage to the computation stage, which significantly improves the circuit utilization. Based on large-scale simulations, we demonstrate HLF can significantly reduce the iteration communication time by up to 64.97% compared to the state-of-the-art circuit scheduler Sunflow. Besides, SWRTF can save up to 42.9%, 54.2%, 27.2% of Weighted-Job-Completion-Time (WJCT) compared to Shortest-Job-First, Baraat and Weighted-First inter-job scheduling algorithms, respectively.

Energy- and locality-efficient multi-job scheduling based on MapReduce for heterogeneous datacenter

Job scheduling of MapReduce is a research hot spot, especially on the heterogeneous datacenter. Huge energy consumption and operating costs are key challenges. Most of the previous work only considers the scheduling optimization of a single job. In this paper, we take multiple jobs of MapReduce as research objects and focus on the goal of “jointly optimizing the scheduling time, job costs and energy consumption.” For that, an energy- and locality-efficient MapReduce multi-job scheduling algorithm is developed for the heterogeneous datacenter. Firstly, we use rack as the basic unit of resource in job scheduling to reduce data communication between jobs and to facilitate energy savings. Secondly, according to the capacity of heterogeneous rack, we design a multi-job pre-mapping method to optimize the execution order of jobs and jointly optimize the scheduling time, job costs and energy consumption. Based this pre-mapping method, we can assign one job to the virtual machine on the same rack, so as to minimize the amount of online rack. This centralized mapping strategy is very helpful to save energy and reduce data transmission of jobs. Thirdly, the map and reduce tasks of a job will be divided into multiple task groups for parallel execution, thereby further reducing data communication and energy consumption. Finally, a lot of experimental results prove the advantages of our algorithm.

An energy-aware bi-level optimization model for multi-job scheduling problems under cloud computing

Recently, how to reduce huge energy consumption of data centers has caught wide attention in cloud computing. One effective way is to improve the energy efficiency of servers. To achieve this goal, we propose a new energy-aware multi-job scheduling model based on MapReduce in this paper. In the proposed model, first, the variation of energy consumption with the performance of servers is taken into account; second, since network bandwidth is a relatively limited resource in cloud computing, 100 % data locality is guaranteed; last but not least, considering that task-scheduling strategies depend directly on data placement policies, we formulate the problem as an integer bi-level programming model. It is worth noticing that there are usually tens of thousands of tasks to be scheduled in the cloud, so this is a large-scale optimization problem. In order to solve it efficiently, a local search operator is specifically designed, based on which, a bi-level genetic algorithm is proposed in this paper. Finally, numerical experiments indicate the effectiveness of the proposed model and algorithm.

Energy-Efficient PSO and Latency Based Group Discovery Algorithm in Cloud Scheduling

Cloud computing is a large model change of computing system. It provides high scalability and flexibility among an assortment of on-demand services. To imporve the performance of the multi-cloud environment in distributed application might require less energy efficiency and minimal inter-node latency correspondingly. The major problem is that the energy efficiency of the cloud computing data center is less if the number of server is low, else it increases. To overcome the energy efficiency and network latency problem a novel energy-efficient particle swarm optimization representation for multi-job scheduling and Latency representation for the grouping of nodes with respect to network latency is proposed. The scheduling procedure is through on the basis of network latency and energy efficiency. Scheduling schema is the main part of Cloud Scheduler component, which helps the scheduler in scheduling decision on the base of dissimilar criterion. It also works well with incomplete latency information and performs intelligent grouping on the basis of both network latency and energy efficiency. Design a realistic particle swarm optimization algorithm for the cloud servers and construct an overall energy competence based on the purpose of the servers and calculation of fitness value for each cloud servers. Also, in order to speed up the convergent speed and improve the probing aptitude of our algorithm, a local search operative is introduced. Finally, the experiment demonstrates that the proposed algorithm is effectual and well-organized.

Improving Performance in Cloud using Multi-Job Scheduling based Group Discovery Algorithm

computing is a large model change of computing system. It provides high scalability and flexibility among an assortment of on-demand services. To imporve the performance of the multi-cloud environment in distributed application might require less energy efficiency and minimal inter-node latency correspondingly. The major problem is that the energy efficiency of the cloud computing data center is less if the number of server is low, else it increases. To overcome the energy efficiency and network latency problem a novel energy-efficient particle swarm optimization representation for multi-job scheduling and Latency representation for the grouping of nodes with respect to network latency is proposed. Design a realistic particle swarm optimization algorithm for the cloud servers and construct an overall energy competence based on the purpose of the servers and calculation of fitness value for each cloud servers. Also, in order to speed up the convergent speed and improve the probing aptitude of our algorithm, a local search operative is introduced. Finally, the experiment demonstrates that the proposed algorithm is effectual and well-organized.

Resource intensity aware job scheduling in a distributed cloud

A Dominant Resource Fairness (DRF) based scheme for job scheduling in distributed cloud computing systems which was modeled as multi-job scheduling and multi-resource allocation coupling problem is proposed, where the resource pool is constructed from a large number of distributed heterogeneous servers, representing different points in the configuration space of resources such as processing, memory, storage and bandwidth. By introducing dominant resource share of jobs and virtual machines, the multi-job scheduling and multi-resource allocation joint mechanism significantly improves the cloud system's resource utilization, yet with a substantial reduction of job completion times. We show through experiments and case studies the superior performance of the algorithms in practice.

A new multi-objective bi-level programming model for energy and locality aware multi-job scheduling in cloud computing

How to reduce power consumption of data centers has received worldwide attention. By combining the energy-aware data placement policy and locality-aware multi-job scheduling scheme, we propose a new multi-objective bi-level programming model based on MapReduce to improve the energy efficiency of servers. First, the variation of energy consumption with the performance of servers is taken into account; second, data locality can be adjusted dynamically according to current network state; last but not least, considering that task-scheduling strategies depend directly on data placement policies, we formulate the problem as an integer bi-level programming model. In order to solve the model efficiently, specific-design encoding and decoding methods are introduced. Based on these, a new effective multi-objective genetic algorithm based on MOEA/D is proposed. As there are usually tens of thousands of tasks to be scheduled in the cloud, this is a large-scale optimization problem and a local search operator is designed to accelerate convergent speed of the proposed algorithm. Finally, numerical experiments indicate the effectiveness of the proposed model and algorithm.