Building a fault tolerant framework with deadline guarantee in big data stream computing environments

Abstract

Big data stream computing systems should work continuously to process streams of on-line data. Therefore, fault tolerance is one of the key metrics of quality of service in big data stream computing. In this paper, we propose a fault tolerant framework with deadline guarantee for stream computing called FTDG. First, FTDG identifies the critical path of a data stream graph at a given data stream throughput, and quantifies the system reliability of a data stream graph. Second, FTDG allocates tasks by the fault tolerance aware heuristic and critical path scheduling mechanism. Third, FTDG online optimizes the task scheduling by reallocating the critical vertices on the critical path of the data stream graph to lower the response time and reduce system fluctuations. Theoretical as well as experimental results demonstrate that the FTDG makes a desirable trade-off between high fault tolerance and low response time objectives in big data stream computing environments.