Job Scheduling System Research Articles

An ANN Model to Predict Select Performance Measures for Job Scheduling System

The purpose of this work is to design and develop an artificial neural network (ANN) model for prediction of the select performance measures (PMs) of a single machine job scheduling (SMJS) system, under the influence of varying levels of input variables. Ten thousand scheduling problems were randomly generated and for each problem values of five PMs were computed which were used to train, test, and validate different ANN models in order to develop the optimal ANN model for accurate prediction of the PMs for a given set of input variables. An ANN model with two hidden layers having 16 neurons in each hidden layer was found to be the optimum model for prediction of the PMs as it resulted in the minimum mean squared error. The actual and predicated values of the PMs obtained from the optimum ANN model were compared and it was found that they were in close agreement. The ANN model developed in this study may be used by the managers, practitioners and other decision makers to define appropriate values of input variables that will yield better PMs. Further, the approach used in the study to develop ANN model may be used for development of similar ANN models for other manufacturing systems. This study provides an ANN-based prediction model to predict PMs of the single machine job scheduling system which may however, be modified as per the requirements of other manufacturing systems to predict their PMs.

Machine Learning Feature Based Job Scheduling for Distributed Machine Learning Clusters

With the rapid proliferation of Machine Learning (ML) and Deep learning (DL) applications running on modern platforms, it is crucial to satisfy application performance requirements such as meeting deadline and ensuring accuracy. To this end, researchers have proposed several job schedulers for ML clusters. However, none of the previously proposed schedulers consider ML model parallelism, though it has been proposed as an approach to increase the efficiency of running large-scale ML and DL jobs. Thus, in this paper, we propose an ML job Feature based job Scheduling system (MLFS) for ML clusters running both data parallelism and model parallelism ML jobs. MLFS first uses a heuristic scheduling method that considers an ML job’s spatial and temporal features to determine task priority for job queue ordering in order to improve job completion time (JCT) and accuracy performance. It uses the data from the heuristic scheduling method for training a deep reinforcement learning (RL) model. After the RL model is well trained, it then switches to the RL method to automatically make decisions on job scheduling. In addition, MLFS has a system load control method that selects tasks from overloaded servers to move to underloaded servers based on task priority, and also intelligently removes the tasks that generate little or no improvement on the desired accuracy performance when the system is overloaded to improve JCT and accuracy by job deadline. Furthermore, we propose Optimal ML iteration stopping method that determines the proper time to stop training ML model when this model reaches the minimum loss value. Our real experiments and large-scale simulation based on real trace show that MLFS reduces JCT by up to 53% and makespan by up to 52%, and improves accuracy by up to 64% when compared with existing ML job schedulers. We also open sourced our code.

中国SKA区域中心原型系统——软件平台

The Square Kilometre Array (SKA) radio telescope is designed to revolutionize scientific breakthroughs in a variety of scientific fields, and the SKA software system is one of the key factors influencing scientific products. The software environment for processing SKA science data must be versatile, flexible, and adaptable. The SKA Regional Centre (SRC) serves as a platform for astronomers to analyze SKA data, conduct scientific research, and interact with other academics. To enable automated parallel processing of observational data from various scientific fields, Chinese scientists have developed the China SRC-prototype (CSRC-P), installed a job scheduling system widely used by large supercomputers, installed an astronomical software platform capable of processing observational data from current leading radio telescopes, and deployed multiple scientific data processing pipelines. This paper describes the software platform of the CSRC-P as well as pipelines for processing SKA precursor telescope data, such as the low-frequency continuum imaging pipeline, spectral line imaging pipeline, and very long baseline interferometry data processing pipeline. Users worldwide have successfully conducted scientific research on SKA using this platform. The knowledge gained from the construction and operation of this platform will be useful in constructing a full-scale SRC in the future.

InterARTIC: an interactive web application for whole-genome nanopore sequencing analysis of SARS-CoV-2 and other viruses.

MotivationInterARTIC is an interactive web application for the analysis of viral whole-genome sequencing (WGS) data generated on Oxford Nanopore Technologies (ONT) devices. A graphical interface enables users with no bioinformatics expertise to analyze WGS experiments and reconstruct consensus genome sequences from individual isolates of viruses, such as SARS-CoV-2. InterARTIC is intended to facilitate widespread adoption and standardization of ONT sequencing for viral surveillance and molecular epidemiology.ResultsWe demonstrate the use of InterARTIC for the analysis of ONT viral WGS data from SARS-CoV-2 and Ebola virus, using a laptop computer or the internal computer on an ONT GridION sequencing device. We showcase the intuitive graphical interface, workflow customization capabilities and job-scheduling system that facilitate execution of small- and large-scale WGS projects on any common virus.Availability and implementationInterARTIC is a free, open-source web application implemented in Python that executes best-practice command line workflows from the ARTIC network. The application can be downloaded as a set of pre-compiled binaries that are compatible with all common Linux distributions, Windows with Linux subsystems, MacOSX and ARM systems. All code can be found on GitHub at https://github.com/Psy-Fer/interARTIC/ and documentation can be found at https://github.com/Psy-Fer/interARTIC/.Supplementary information Supplementary data are available at Bioinformatics online.

Scheduling of workflow jobs based on twostep clustering and lowest job weight

AbstractScheduling is an important technique to improve the productivity of workflow applications in a grid system. Serving workflow applications added overhead on the scheduling system since the scheduler should choose the best zero‐dependent job that improves the performance of a grid system. This paper proposes two dependent job scheduling algorithms based on analyzing the effect of categorical and continuous variables of the job, which are used in the calculation of job weight. To find the job weight, twostep clustering is used with 10 groups and the ranking equation. In addition, to verify the ability to apply the proposed algorithms in a real environment, weighted least squares estimation is used. The results showed that the prediction rate is equal to 99.88%, which indicates that the proposed algorithms could be applied in a real grid system with low overhead. Through simulations and after testing the proposed algorithms, the average results showed that the Dependent Job Scheduling (DJS) algorithm outperformed the previous algorithms, in total execution time and an average waiting time with an improvement value of 1.18 and 1.92 times, respectively. While DJS algorithm with weighting factor (DJSJP) outperformed the previous algorithms in total execution time only with an improvement value equal to 1.17 times. The overall results indicated that the proposed algorithms are efficient to be used in a grid system, besides four bits are sufficient to improve the performance of the job scheduling system.

ELS: Emulation system for debugging and tuning large-scale parallel programs on small clusters

Continuous scaling-up of high-performance computing systems has brought challenges to the debugging and tuning of large-scale parallel programs. Firstly, to locate bugs in a program or tune its performance, programmer often needs to execute the program in a specified scale repeatedly, which consumes massive resources; secondly, due to the extensively used job scheduling systems, programmers can only submit their programs as jobs and cannot interact with them, which restricts debugging efficiency and flexibility. To address these challenges, this paper proposes an emulation system that supports debugging and tuning of large-scale parallel programs by executing parallel programs in the desired scale on a small cluster. The program is firstly executed in the desired scale on the target HPC system to record necessary information; then, programmers can choose and re-execute a subset of processes of the program repeatedly on a small cluster, during which the emulation system controls the execution of the processes, and programmers can debug their programs by attaching tools to the selected processes. Moreover, our system supports popular CPU+GPU heterogeneous architecture. The system is evaluated on a small cluster, while a 1000-node system is used as the target HPC system; experimental results demonstrate the accuracy and efficiency of emulation-execution.

Construction and Management of Astronomical Big Data Analysis and Multi-dimensional Information Visualization Platform

The development of modern astronomy is rapidly and astronomical data increases exponentially. The HPC architecture based on GPU provides an efficient way of astronomic big data computing. Based on secure Ipv6 network environment, PMO has constructed the Big Data Analysis and Multi-dimensional Information Visualization Platform, which can reach the peak computing speed of 352Tflops and the totally storage capacity of 288TB. The platform is composed of 25 computing nodes, one management node and 5 storage nodes. The use of user-friendly, centralized cluster management software, the deployment of proprietary environmental control settings and multi-dimensional visualization of safety management systems form a multi-level, tridimensional and efficient management structure. A high-speed, high-capacity, highly reliable, secure and efficient high-performance computing cluster is finally constructed. The platform has a fully redundant, self-healing, highly scalable distributed storage system, a high-performance InfiniBand parallel computing and storage network, a complete job scheduling system, a cuda parallel computing architecture, and a variety of physical software tools for astronomical applications. It offers a great help to astronomical research topics such as astronomical image processing, moving target extraction, space target orbit calculation, numerical cosmology, cosmology simulation, galaxy fluid simulation. Thus it provides an important information support for the research work of 3 major breakthroughs and 5 key cultivation directions in the "One Three Five" plan of Purple Mountain Observatory.

Design tools for electromagnetic-driven multi-physics systems using high performance computing

Purpose The design of electromagnetic systems for a variety of applications such as induction heating, electrical machines, actuators and transformers requires the solution of a multi-physics problem often involving thermal, structural and mechanical coupling to the electromagnetic system. This results in a complex analysis system embedded within an optimization process. The appearance of high-performance computing systems over the past few years has made coupled simulations feasible for the design engineer. When coupled with surrogate modelling techniques, it is possible to significantly reduce the wall clock time for generating a complete design while including the impact of the multi-physics performance on the device. Design/methodology/approach An architecture is proposed for linking multiple singe physics analysis tools through the material models and a controller which schedules the execution of the various software tools. The combination of tools is implemented on a series of computational nodes operating in parallel and creating a “super node” cluster within a collection of interconnected processors. Findings The proposed architecture and job scheduling system can allow a parallel exploration of the design space for a device. Originality/value The originality of the work derives from the organization of the parallel computing system into a series of “super nodes” and the creation of a materials database suitable for multi-physics interactions.

Design of Job Scheduling System and Software for Packaging Process with SPT, EDD, LPT, CDS and NEH algorithm at PT. ACP

PT. Avesta Continental Pack is a manufacturer of flexible packaging for pharmaceutical products. The company faces ineffective production scheduling and repeatedly failed to finish orders at the expected time limit. Customers requesting orders in different finishing time contributes in causing delays in the delivery process. Consequently, customers ended up claiming the fee because of the late delivery. Therefore, a more advanced scheduling system will assist the company in managing the delivery time. The purpose of this research is to establish the most effective job scheduling arrangement to minimize mean tardiness as well as mean lateness job. As the production process uses flow shop method, this research applies heuristic methods, such as Earliest Due Date (EDD) method, Short Processing Time (SPT), Nawaz, Enscore, and Ham (NEH) algorithm and Campbell, Dudek, and Smith (CDS) algorithm. The result shows that SPT method provides the most optimum outcome, specifically by reducing mean tardiness by 801.81 minutes or 85.57% and decreasing mean lateness job by 2 or 66.67%. The best job sequence is subsequently job 8 - job 4 - job 1 - job 6 - job 7 - job 5 - job 2 - job 3, which minimizes the average delay time from 937.05 minutes to 135.24 minutes.

Using The Lisa Genetic Cluster

BackgroundThrough the PGC-DAC portal PGC users obtain access to PGC-related phenotypic and genetic data. All PGC data is stored on the Dutch Genetic Cluster Computer, called LISA. PGC users sign a contract in which they state they will not remove the data to which they have access to from the LISA cluster. All analyses with the data thus have to be carried out on the LISA cluster. MethodsThe session will show how to run analyses on the LISA cluster, including how to write jobs efficiently, how to submit jobs, how to manage jobs, and how to store data efficiently. It will also explain the rationale behind the job scheduling and priority system. ResultsN/A (education day) ConclusionsThe aim of this session is to inform PGC members how to handle and analyze the PGC data on the LISA cluster in an efficient and responsible way.

Scheduling nurses’ shifts at PGI Cikini Hospital

Hospitals play an essential role in the community by providing medical services to the public. In order to provide high quality medical services, hospitals must manage their resources (including nurses) effectively and efficiently. Scheduling of nurses’ work shifts, in particular, is crucial, and must be conducted carefully to ensure availability and fairness. This research discusses the job scheduling system for nurses in PGI Cikini Hospital, Jakarta with Goal Programming approach. The research objectives are to identify nurse scheduling criteria and find the best schedule that can meet the criteria. The model has hospital regulations (including government regulations) as hard constraints, and nurses’ preferences as soft constraints. We gather primary data (hospital regulations and nurses’ preferences) through interviews with three Head Nurses and distributing questionnaires to fifty nurses. The results show that on the best schedule, all hard constraints can be satisfied. However, only two out of four soft constraints are satisfied. Compared to current scheduling practice, the resulting schedule ensures the availability of nurses as it satisfies all hospital’s regulations and it has a higher level of fairness as it can accommodate some of the nurses’ preferences.

EUPAN enables pan-genome studies of a large number of eukaryotic genomes.

Pan-genome analyses are routinely carried out for bacteria to interpret the within-species gene presence/absence variations (PAVs). However, pan-genome analyses are rare for eukaryotes due to the large sizes and higher complexities of their genomes. Here we proposed EUPAN, a eukaryotic pan-genome analysis toolkit, enabling automatic large-scale eukaryotic pan-genome analyses and detection of gene PAVs at a relatively low sequencing depth. In the previous studies, we demonstrated the effectiveness and high accuracy of EUPAN in the pan-genome analysis of 453 rice genomes, in which we also revealed widespread gene PAVs among individual rice genomes. Moreover, EUPAN can be directly applied to the current re-sequencing projects primarily focusing on single nucleotide polymorphisms. EUPAN is implemented in Perl, R and C ++. It is supported under Linux and preferred for a computer cluster with LSF and SLURM job scheduling system. EUPAN together with its standard operating procedure (SOP) is freely available for non-commercial use (CC BY-NC 4.0) at http://cgm.sjtu.edu.cn/eupan/index.html . ccwei@sjtu.edu.cn or jianxin.shi@sjtu.edu.cn. Supplementary data are available at Bioinformatics online.

Load Balancing In Decentralized Grid Scheduling Systems Using Bee Colony Algorithm

Load balancing is an important part of grid scheduling systems which aims to utilize dynamic and decentralized grid resources without any overcapacity happening at any resources. Load balancing is being considered as an important phase for distributed computing environments. In large scale distributed systems such as grid/cloud computing, it is necessary to use a distributed load balancing system in job scheduling system. Distributed load balancing was identified as a major concern to allow grid computing to scale up the job scheduling. Many algorithms had been proposed for finding the solution of load balancing problem in these fields. But very few algorithms are proposed for distributed load balancing in grid computing environments. In this paper, we have developed an adaptable artificial bee colony (ABC) algorithm for dynamic dividing and scheduling jobs via decentralized schedulers. The proposed method significantly enhances the load balance on the resources. Besides, this method does not need primary information about the capacity and power of the resources which is compatible with the variable and heterogeneous nature of decentralized scheduling. The proposed method is compared with the cuckoo based scheduling algorithm and OSL decentralized scheduling method. The results of the simulations revealed that the proposed method is significantly better than the OSL method. It also outperforms the cuckoo based scheduling algorithm in most of simulations.

Fuxi

Scalability and fault-tolerance are two fundamental challenges for all distributed computing at Internet scale. Despite many recent advances from both academia and industry, these two problems are still far from settled. In this paper, we present Fuxi, a resource management and job scheduling system that is capable of handling the kind of workload at Alibaba where hundreds of terabytes of data are generated and analyzed everyday to help optimize the company's business operations and user experiences. We employ several novel techniques to enable Fuxi to perform efficient scheduling of hundreds of thousands of concurrent tasks over large clusters with thousands of nodes: 1) an incremental resource management protocol that supports multi-dimensional resource allocation and data locality; 2) user-transparent failure recovery where failures of any Fuxi components will not impact the execution of user jobs; and 3) an effective detection mechanism and a multi-level blacklisting scheme that prevents them from affecting job execution. Our evaluation results demonstrate that 95% and 91% scheduled CPU/memory utilization can be fulfilled under synthetic workloads, and Fuxi is capable of achieving 2.36T-B/minute throughput in GraySort. Additionally, the same Fuxi job only experiences approximately 16% slowdown under a 5% fault-injection rate. The slowdown only grows to 20% when we double the fault-injection rate to 10%. Fuxi has been deployed in our production environment since 2009, and it now manages hundreds of thousands of server nodes.

Job Scheduling for Cloud Computing Using Neural Networks

Cloud computing aims to maximize the benefit of distributed resources and aggregate them to achieve higher throughput to solve large scale computation problems. In this technology, the customers rent the resources and only pay per use. Job scheduling is one of the biggest issues in cloud computing. Scheduling of users’ requests means how to allocate resources to these requests to finish the tasks in minimum time. The main task of job scheduling system is to find the best resources for user’s jobs, taking into consideration some statistics and dynamic parameters restrictions of users’ jobs. In this research, we introduce cloud computing, genetic algorithm and artificial neural networks, and then review the literature of cloud job scheduling. Many researchers in the literature tried to solve the cloud job scheduling using different techniques. Most of them use artificial intelligence techniques such as genetic algorithm and ant colony to solve the problem of job scheduling and to find the optimal distribution of resources. Unfortunately, there are still some problems in this research area. Therefore, we propose implementing artificial neural networks to optimize the job scheduling results in cloud as it can find new set of classifications not only search within the available set.

A Novel Cloud Computing Service Job Scheduling Optimization Model

In this paper, we put forward a differential service-oriented and self-adaptive job scheduling system in Cloud Computing environment. Analysis and number results show that our approach for job scheduling system can not only guarantee the QoS requirements of the CCUs jobs, but also can make the maximum profits for the CCSP. To the best of our knowledge, the job scheduling system in our paper maybe the first paper to consider both the QoS requirement of jobs and Cloud Computing service providers profits.

시간가치와 중요도 기반의 분류 작업 스케쥴링

정보가 폭주하고 해야할 일들이 많은 상황에서 바쁜 현대인에게 있어서는 효율적인 에너지와 시간관리가 매우 중요한 일로 여겨지고 있다. 이미 효율적인 시간관리 스케쥴링 방식은 성공적인 삶을 살기 위한 매우 중요한 요소가 되었기 때문에 사건중심의 기존의 단순한 스케쥴링 방식보다 더 정교하게 이러한 기능을 지원해 줄 수 있는 지능적 시스템이 필요하다. 따라서 본 논문에서는 작업을 시간가치와 중요도에 따라 분류하고 이를 효율적으로 시간 프레임에 할당할 수 있는 분류 작업 스케쥴링 방법을 제안하고자 한다. 특히 시간가치가 높은 작업을 우선적으로 처리하고 동시에 시간가치는 높지 않으나 미래의 성공의 열쇠를 쥐고 있는 매우 중요한 일들을 꾸준히 수행할 수 있도록 시간 프레임에 일정부분을 고정 할당하는 새로운 방법을 제안한다.

A Reliable and Fault Tolerant Job Scheduling System in Market-Based Grids Using Case-Based Reasoning Method

In market-based Grid systems, a main aim is to execute jobs with considered quality of service requirements based on user defined budget. Since grid has heterogonous resource with unpredictable faults, the user cost constraints and expected service requirements may not provided. Therefore, using a better approach to resource scheduling to reduce fault is necessary. This paper presents a predictive approach on fault tolerance mechanisms for faultless job scheduling on market-based grids. The Case-Based Reasoning technique has been used for selecting fault tolerant nodes. This approach applies a specific structure in order to prepare fault tolerance between provider nodes to retain system in a safe state with minimum data transferring. Certainly, this algorithm increases fault tolerant confidence therefore, performance of grid will be high.

Grouping Based Job Scheduling Algorithm Using Priority Queue and Hybrid Algorithm in Grid Computing

Grid computing enlarge with computing platform which is collection of heterogeneous computing resources connected by a network across dynamic and geographically dispersed organization to form a distributed high performance computing infrastructure. Grid computing solves the complex computing problems amongst multiple machines. Grid computing solves the large scale computational demands in a high performance computing environment. The main emphasis in the grid computing is given to the resource management and the job scheduler .The goal of the job scheduler is to maximize the resource utilization and minimize the processing time of the jobs. Existing approaches of Grid scheduling doesn’t give much emphasis on the performance of a Grid scheduler in processing time parameter. Schedulers allocate resources to the jobs to be executed using the First come First serve algorithm. In this paper, we have provided an optimize algorithm to queue of the scheduler using various scheduling methods like Shortest Job First, First in First out, Round robin. The job scheduling system is responsible to select best suitable machines in a grid for user jobs. The management and scheduling system generates job schedules for each machine in the grid by taking static restrictions and dynamic parameters of jobs and machines into consideration. The main purpose of this paper is to develop an efficient job scheduling algorithm to maximize the resource utilization and minimize processing time of the jobs. Queues can be optimized by using various scheduling algorithms depending upon the performance criteria to be improved e.g. response time, throughput. The work has been done in MATLAB using the parallel computing toolbox.

Development of Job Scheduling and Machine Loading System in FMS

Development of Job Scheduling and Machine Loading System in FMS