Cost effective stream workflow scheduling to handle application structural changes

Abstract

Stream workflow is a network of big data streaming applications that acts as key enabler for real-time analysis from Internet of Things data. Smart traffic management and smart grid are examples of stream workflow. The focus of existing work is on streaming operator graphs which differs from stream workflow and handling data fluctuations without significant consideration of different dynamic forms that could happen in the structure of data pipelines. This paper investigates the scheduling problem of stream workflow to support runtime alterations of stream workflow deployment, so that scheduling plans will be revised to handle stream workflow applications with continuously changing characteristics. It proposes a pluggable dynamic scheduling technique that accepts user-defined algorithms to handle stream workflow runtime changes. It also presents three different plug-in algorithms and methods to enable auto-scaling of this workflow in a Multicloud environment. The experimental results of the quality of the solution showed that the proposed plug-in optimisation technique is more efficient than baseline and dynamic fair-share techniques to handle runtime changes.