Placement of distributed stream processing over heterogeneous infrastructures

Abstract

Data Stream Processing (DSP) applications can extract, in a timely manner, valuable information from distributed data sources (e.g., sensing devices, social networks). These applications are subject to unpredictable and varying workloads and have to satisfy strict quality requirements, usually expressed in terms of latency, availability, and throughput. To successfully execute DSP applications, recent trends investigate the possibility of exploiting decentralized computing resources, which nonetheless pose new challenges due to the network and system heterogeneity, geographic distribution, and non-negligible network latencies.The doctorate work, presented in this paper, investigates the deployment of DSP applications with Quality of Service (QoS) requirements over a distributed infrastructure of heterogeneous computing and networking resources. Specifically, to support our study, we extend an open-source DSP system, Apache Storm, by providing mechanisms for executing distributed QoS-aware placement policies and self-adaptation. Then, we provide a general formulation of the optimal placement problem for DSP applications, modeling the heterogeneity of the execution environment.The ongoing research aims at investigating the following directions. First, we will design heuristics able to determine the best placement in a feasible amount of time. Second, we will investigate runtime adaptation strategies and online placement algorithms. Third, to prove the generality of our approach, we will customize the designed solutions for similar problems (e.g., service selection, container deployment).