Complex Event Processing under Constrained Resources by State-Based Load Shedding

Abstract

Complex event processing (CEP) systems evaluate queries over event streams for low-latency detection of user-specified event patterns. They need to process streams of growing volume and velocity, while the heterogeneity of event sources yields unpredictable input rates. Against this background, models and algorithms for the optimisation of CEP systems have been proposed in the literature. However, when input rates grow by orders of magnitude during short peak times, exhaustive real-time processing of event streams becomes infeasible. CEP systems shall therefore resort to best-effort query evaluation, which maximises the accuracy of pattern detection while staying within a predefined latency bound. For traditional data stream processing, this is achieved by load shedding that drops some input data without processing it, guided by the estimated importance of particular data entities for the processing accuracy. In this work, we argue that such input-based load shedding is not suited for CEP queries in all situations. Unlike for relational stream processing, where the impact of shedding is assessed based on the operator selectivity, the importance of an event for a CEP query is highly dynamic and largely depends on the state of query processing. Depending on the presence of particular partial matches, the impact of dropping a single event can vary drastically. Hence, this PhD project is devoted to state-based load shedding that, instead of dropping input events, discards partial matches to realise best-effort processing under constrained resources. In this paper, we describe the addressed problem in detail, sketch our envisioned solution for state-based load shedding, and present preliminary experimental results that indicate the general feasibility of our approach.