Elasticity management of Streaming Data Analytics Flows on clouds

Abstract

In this paper, we present a framework for resource management of Streaming Data Analytics Flows (SDAF). Using advanced techniques in control and optimization theory, we design an adaptive control system tailored to the data ingestion, analytics, and storage layers of the SDAF that is able to continuously detect and self-adapt to workload changes for meeting the users' service level objectives. Our experiments based on a real-world SDAF show that, the proposed control scheme is able to reduce the deviation from desired utilization of resources by up to 48% compared to existing techniques.