Queue Waiting Time Prediction for Large-scale High-performance Computing System

Abstract

Traditionally, high-performance computing (HPC) systems have been extensively utilized in many science fields including big data analysis and machine learning. Such large-scale HPC resources typically use the queue management systems which prefer space-sharing method to allocate resources. In space-sharing method, it is natural that a queue waiting time occurs until the resources are available if resources are not sufficient. When the predicted information on such a waiting time is provided, it is possible to improve the performance of scheduler. For this, in this paper, we propose a prediction method of queue waiting time based on the job log file created in the HPC system actually under operation. The prediction technique presented in this paper largely comprises three phases. The first phase is a pre-processing of data in constant time intervals. In the second phase, major features are selected through a factor analysis and clustering is conducted based on the selected features. In the third phase, a waiting time of the next job is predicted using the sliding window method based on the jobs that were clustered.