Elastic stateful stream processing in storm

Abstract

The advent of the Big Data era and the diffusion of Cloud computing have renewed the interest in Data Stream Processing (DSP) applications, which can timely extract useful information from distributed data sources. Due to the unpredictable rate at which the sources may produce data, DSP applications demand high dynamism. Storm has emerged as a widely adopted DSP system, which, although having many desirable features, shows some limitations due to the lack of adaptation capabilities. In this paper, we extend Storm with two mechanisms that support the run-time adaptation of DSP applications. Specifically, we introduce new components that allow automatic elasticity and stateful migration of the application components. The experimental results show the benefits of the newly introduced functionalities that, albeit equipped with proof of concept policies, allow to properly cope with workload variations while improving the resource utilization of the underlying infrastructure.