Complex Event Processing Operators Research Articles

ACEP: an adaptive strategy for proactive and elastic processing of complex events

The execution of complex event processing (CEP) applications on a set of clustered homogenous computing nodes is latency-sensitive, especially when workload conditions widely change at runtime. To manage the varying workloads of nodes in a scalable and cost-effective manner, adjusting the application parallelism at runtime is critical. To tackle the scalability challenge, we have extended an existing parallelization model called PARS that only supports stateless CEP operators and runs operators in parallel regions without changing the number of computing nodes assigned to parallel regions. We have added new features to PARS in support of stateful operators by introducing local controllers and new initiator and terminator event types, making partitioning fully transparent to application developers. We have proved the correctness of this extended model, called PARS+ , with respect to its presented formal definition. We have then used PARS+ as the base parallelization model in formulation of an adaptive strategy called ACEP to auto-scale operators including the stateful operators. Scaling decisions are governed by a predictive performance model that uses a control-theoretic method for estimating the resource and latency costs of each operator at runtime. The loads of clustered compute nodes are monitored, and compute nodes in a parallel region are reconfigured at runtime to ensure a balanced load on all compute nodes, accruing minimum cost to parallelize a stateful operator. ACEP minimizes network delays because it does not force using the shared state or techniques that employ state migration. We have built an event generator to simulate event sources and experimentally evaluate the ACEP strategy in terms of response time and resource costs. Two variant implementations of ACEP have been compared with the elastic strategy presented by Xiao et al. (2018), and the findings demonstrated that our implementations had adapted themselves better in different resource and response-time-sensitive scenarios, with lower response time (6%) and lower resource cost (8%).

Network-wide complex event processing over geographically distributed data sources

In this paper we focus on Complex Event Processing (CEP) applications where the data is generated by sites that are geographically dispersed across large regions. This geographic distribution, combined with the size of the collected data, imposes severe communication and computation challenges. To attack these challenges, we propose a novel approach for geographically distributed CEP, which combines algorithmic and systems contributions. At an algorithmic level, our work combines an in-network processing approach, which pushes parts of the processing (i.e., CEP operators) towards the sources of their input events, along with a push–pull paradigm, in order to reduce the amount of communicated events. We present optimal (but computationally expensive) solutions which seek to minimize the maximum bandwidth consumption given input latency constraints for detecting events, as well as efficient greedy and heuristic algorithmic variations for our problem. At a systems level, we explain how existing CEP engines can support, with minimal modifications, our algorithms. Our experimental evaluation, using mainly real datasets and network topologies, demonstrates that the power of our techniques lies in the combination of the in-network with the push–pull paradigm, thus allowing our algorithms to significantly outperform related centralized push–pull or conventional in-network processing approaches.