Large-Scale Data Intensive Heterogeneous Task Scheduling Method Based on Parallel GATS-TS Algorithm

Abstract

Data-intensive applications need to process ever-increasing massive data, which puts forward high requirements on the performance of computing resources. Traditional scheduling methods can no longer adapt to the randomness of heterogeneous tasks, so a large-scale data-intensive heterogeneous task scheduling method based on parallel GATS-TS algorithm is proposed. According to the large-scale data task scheduling strategy, the tasks of the whole workflow are divided into various stages according to the data dependence, and then the task scheduling is carried out step by step. Large-scale data-intensive heterogeneous tasks are classified and the performance of each node is dynamically evaluated according to the different task types. According to the classification results of heterogeneous tasks, the completion time of tasks to be assigned at each stage was determined, and the scheduling model was established by using the parallel GATS- TS algorithm. The fitness function of the algorithm was the reciprocal of the completion time, and the model output the optimization results of intensive heterogeneous task scheduling. The test results show that the design method can shorten the task completion time and reduce the scheduling length ratio, and achieve good scheduling results under the condition of large number of tasks