Recovery-Conscious Adaptive Watermark Generation for Time-Order Event Stream Processing

Abstract

Achieving low latency in time-order real-time event stream processing of data produced by a large number of IoT devices is difficult due to the buffering required for sorting event messages arrived. Despite the existence of real-time IoT applications that highly depend on the order of event occurrences, event messages arriving from the devices to the event processing system are not time-ordered in many cases. Therefore, such an event processing system needs to buffer arriving event messages, sort, and then process them. This buffering tends to be decided as a safe large period to guarantee correctness in terms of time-ordering, and leads to an increase of latency. Furthermore, failure recovery mechanism of a stateful stream processing deployed into distributed processing nodes makes this problem more difficult. It frequently causes large time-order reversal among event messages in the distributed stream processing system, that cannot be handled with a buffering designed for real-time processing during normal periods. In this paper, we introduce a novel technique to adaptively determine the minimum necessary amount of buffering to guarantee time-order sorting depending on the event-time progress in arrival event messages. To be more precise, instead of specifying the actual buffering time, the method adaptively controls the minimum margin to determine the current that shows the event upon which a processing unit in a distributed stream processing system believes to have received all the messages. Further, we developed a complete function (called time alignment) to sort the time-order of messages while keeping low latency in our developed distributed event stream processing platform. Through the experimental evaluations, it is shown that the developed method has the capability to determine a proper watermark to achieve both a strong guarantee on time-order correctness and low latency in real-time event stream processing.