A design of resource fault handling mechanism using dynamic resource reallocation for the resource and job management system

Abstract

Due to the development of NGS technologies and the reduction of analysis cost, it is possible to perform population-scale human genome analysis. Also, large amount of genome data have been exploded recently. It is required for introduction parallel processing using High Performance Computing systems to analyse and handle these large data through genome analysis pipeline. In this paper, we propose the resource fault handling mechanism based on dynamic resource reconfiguration and delayed scheduling for data-intensive pipeline job processing such as genome analysis executed on the large cluster systems interconnected by high speed and low latency network. In order to prevent the abnormal job completion caused by lack of the specific resources, we offer the resource fault detection and handling methods. If the cause of fault is lack of resources, it can be solved by the resource re-allocation and process freezing/resuming based delayed job execution or process migration on the available node.