Work-in-Progress: Incorporating Deadline-Based Scheduling in Tasking Programming Model for Extreme-Scale Parallel Computing

Abstract

Processing and analyzing big data sets updated in real time in an increasing number of applications such as severe weather prediction and particle-physics experiments require the computational power of extreme-scale high-performance computing (HPC) systems. To address the scheduling of massive task/thread sets on these extreme-scale systems, current strategies rely on improving centralized, distributed, and parallel scheduling algorithms as well as virtualization developed for HPC systems which aim to reduce the makespan and balance the load among the computing nodes in these systems. However, these HPC schedulers provide no guarantees on meeting timing constraints such as deadlines that are required in an increasing number of these real-time science workflows. This paper describes a new project which departs from this established trend of best-effort scheduling of large-scale HPC Message Passing Interface (MPI) tasks and ensemble workloads found in fine-grain many-task computing (MTC) applications. The new approach brings real-time scheduling to address the demands of real-time science workloads. This new framework abstracts information about the tasks or threads, and continuously dispatch this workload to meet deadlines and other timing constraints associated with individual tasks or groups of tasks in extreme-scale HPC systems to reduce execution time and energy consumption. This paper introduces deadline-based scheduling in the tasking programming model.