Re-Stream: Real-time and energy-efficient resource scheduling in big data stream computing environments

Abstract

To achieve high energy efficiency and low response time in big data stream computing environments, it is required to model an energy-efficient resource scheduling and optimization framework. In this paper, we propose a real-time and energy-efficient resource scheduling and optimization framework, termed the Re-Stream. Firstly, the Re-Stream profiles a mathematical relationship among energy consumption, response time, and resource utilization, and obtains the conditions to meet high energy efficiency and low response time. Secondly, a data stream graph is modeled by using the distributed stream computing theories, which identifies the critical path within the data stream graph. Such a methodology aids in calculating the energy consumption of a resource allocation scheme for a data stream graph at a given data stream speed. Thirdly, the Re-Stream allocates tasks by utilizing an energy-efficient heuristic and a critical path scheduling mechanism subject to the architectural requirements. This is done to optimize the scheduling mechanism online by reallocating the critical vertices on the critical path of a data stream graph to minimize the response time and system fluctuations. Moreover, the Re-Stream consolidates the non-critical vertices on the non-critical path so as to improve energy efficiency. We evaluate the Re-Stream to measure energy efficiency and response time for big data stream computing environments. The experimental results demonstrate that the Re-Stream has the ability to improve energy efficiency of a big data stream computing system, and to reduce average response time. The Re-Stream provides an elegant trade-off between increased energy efficiency and decreased response time effectively within big data stream computing environments.