Explore unlabeled big data learning to online failure prediction in safety-aware cloud environment

Abstract

Proactive fault management is an important problem in many areas of data management, including cloud computing, big data, vision, machine learning and especially for the cross-domain research of distributed computing and AI (Artificial Intelligence). Unfortunately, most real-world online failure prediction is facing the problem that the used data are difficult to label although the failure prediction should be a supervised learning problem. We observe that, in many cases, the large-scale unlabeled data can be classified through feature extraction and clustering for available prediction, and thus ideas from their combination can be brought to bear. Based on this, we have proposed an online failure prediction framework approach UDFP (Unlabeled Data based online Failure Prediction). It introduces the clustering analysis method based on the combination of the KNN (k-nearest neighbor) and the modularity idea to achieve prediction modeling. It is shown analytically that UDFP can mitigate a supervised learning problem for failure prediction in our situation to some extent. Experimental results demonstrate that UDFP, as a framework approach, has avoided the manual tagging workload and the huge difficulties, improved the predictive accuracy, and reduced cost of data management in safety-aware distributed cloud data centers while enhancing fault-tolerant capabilities and robustness.