Bounding substreams in distributed stream processing

Abstract

A common problem in distributed stream processing is to split a stream into finite chunks of messages (substreams) and to determine their boundaries: stateful streaming operators should clear outdated state; time window operators should release output after all elements within the specified time range have arrived. Most state-of-the-art SPEs use punctuations to divide a stream into bounded substreams of messages. The punctuation approach is powerful but has limitations: it does not support cyclic dataflows, is poorly scalable in some cases due to intensive use of broadcasts, and becomes inefficient when the number of chunks or cluster size becomes significant. We introduce a new substream tracking technique called trAcker that overcomes the limits of punctuations. We experimentally evaluate the properties of trAcker in both synthetic and real-world environments. Experiments show that our technique is scalable, outperforms punctuations for a large number of substreams, and efficiently handles real-world cyclic dataflows.