Task Resource Usage Analysis and Failure Prediction in Cloud

Abstract

To improve the reliability of the cloud computing system it is important to understand the failure characteristics and to predict failures earlier to avoid it. A statistical analysis of workload data on the cloud provides insights into failure characteristics, which can be used as a cue to improve the reliability of the system. This manuscript discusses a statistical analysis of resource usage data of tasks on the large Google cluster dataset, further failure prediction algorithms are developed to predict the failure. Based on the study, it is observed that there is variation in the resource usage pattern, duration of execution and amount of resource used by a failed task as compared to that of a finished task. Different resampling techniques along with XGboost classifier is used to predict the failure of a task in the highly imbalanced dataset and it is observed that Synthetic minority oversampling along with XGboost predicted the task status with precision of 92% and recall of 94.8%.