Self-Adaptive Data Stream Processing in Geo-Distributed Computing Environments

Abstract

An ever increasing number of services requires real-time analysis of collected data streams. Emerging Fog/Edge computing platforms are appealing for such latency-sensitive applications, encouraging the deployment of Data Stream Processing (DSP) systems in geo-distributed environments. However, the highly dynamic nature of these infrastructures poses challenges on how to satisfy the Quality of Service requirements of both the application and the infrastructure providers.In this doctoral work we investigate how DSP systems can face the dynamicity of workloads and computing environments by self-adapting their deployment and behavior at run-time. Targeting geo-distributed infrastructures, we specifically search for decentralized solutions, and propose a framework for organizing adaptation using a hierarchical control approach. Focusing on application elasticity, we equip the framework with decentralized policies based on reinforcement learning. We extend our solution to consider multi-level elasticity, and heterogeneous computing resources. In the ongoing research work, we aim to face the challenges associated with mobility of users and computing resources, exploring complementary adaptation mechanisms.