Scalable complex event processing using adaptive load balancing

Abstract

An essential requirement of large-scale event-driven systems is the real-time detection of complex patterns of events from a large number of basic events and derivation of higher-level events using complex event processing (CEP) mechanisms. Centralized CEP mechanisms are however not scalable and thus inappropriate for large-scale domains with many input events and complex patterns, rendering the horizontal scaling of CEP mechanisms a necessity. In this paper, we propose CCEP as a mechanism for clustering of heterogeneous CEP engines to provide horizontal scalability using adaptive load balancing. We experimentally compare the performance of CCEP with the performances of three CEP clustering mechanisms, namely VISIRI, SCTXPF, and RR. The results of experiments show that CCEP increases throughput by 40 percent and thus it is more scalable than the other three chosen mechanisms when the input event rate changes at runtime. Although CCEP increases the network utilization by about 40 percent, it keeps the load of the system two times more balanced and reduces the input event loss three times.