QoS-aware deployment of data streaming applications over distributed infrastructures

Abstract

The advent of the Big Data era and the diffusion of the Cloud computing paradigm have renewed the interest in Data Stream Processing (DSP) applications, which can timely extract valuable information from an increasing number of data sources (e.g., sensing devices, social networks). The distribution of data sources, the huge and unpredictable volumes of produced data, and the diffusion of heterogeneous computing devices, often available on demand, foster new strategies for an efficient and network-aware usage of the underlying infrastructure. The doctorate work, presented in this paper, investigates the deployment of DSP applications with Quality of Service (QoS) requirements over a large-scale distributed infrastructure of heterogeneous computing and networking resources. Specifically, to support our study, we extend an open-source DSP system, Apache Storm, by providing mechanisms for executing distributed QoS-aware scheduling policies and self-adaptation. We provide a general formulation of the optimal placement problem for DSP application (for short, ODP), taking into account the heterogeneity of computing and networking resources. The ongoing research aims at investigating the following directions. First, we will design heuristics able to determine the best placement in a feasible amount of time. Second, we will investigate runtime adaptation strategies and online placement algorithms. Third, to prove the generality of our approach, we will customize the designed models and algorithms for similar problems (e.g., service selection, container deployment).