Complex Event Processing Systems Research Articles

Cepair: an AI-powered and fog-based predictive CEP system for air quality monitoring

AbstractAir pollution is one of the influential problems threatening the environment and human health today. Therefore, it is critical to develop predictive systems for proactive decisions in solving this problem. Since the prediction of air pollution depends on several complicated factors such as the accuracy of meteorology reports, air pollution accumulation, traffic flow, and industrial emissions, the contribution of historical or real-time predictions to the solution of the problem is limited. To address the existing limitations, we propose a novel AI-powered and Fog-based predictive complex event processing system (CepAIr) for the prediction of future air pollution rates. CepAIr predicts the future air quality of pollutant gases using RNN, LSTM, CNN, and SVR models. Then, it sends the prediction results to decision-makers in an understandable format, enabling them to take proactive actions. Finally, we evaluate the performance of the CepAIr with SVR and DL models. Additionally, we examine CepAIr in terms of end-to-end network delay and measure its impact on the network. The extensive simulation results demonstrate that the CepAIr predicts future pollutant gas concentrations with DL models (especially with CNN) with a high success rate while guaranteeing minimum end-to-end network delay.

Rule based complex event processing for IoT applications: Review, classification and challenges

AbstractIn recent decades, data aggregation and correlation have emerged as a significant and challenging area of research for finding useful relationships between different features of streaming data. Similarly, complex event processing (CEP) has emerged as a vital tool for aggregating and finding patterns from streaming data such as video streaming, real‐time stock trades, sensor data and more. In this context, rule‐based classifier algorithms are widely employed to discover valuable patterns from streaming data and cause‐and‐effect relationships between events. However, streaming data is dynamic and complex in nature, and it is not feasible to update the rules manually by domain experts continuously. On the contrary, dynamic data sometimes fails to adopt rules, which leads to sub optimal CEP implementation. Rule‐based CEP systems come up with their own set of challenges, which primarily includes rules adaptability for streaming IoT data. In dynamic environment, shifting patterns in data streams can impact the performance of the CEP system, as well as the correlation between events, and finding useful patterns is very much challenging. In this article, our objective is to provide a comprehensive survey for CEP using rule‐based algorithms applied across various domain and applications. We present a broad literature review using a sequential expansion of methods used for CEP, which primarily include event producers, preprocessing of events, robust rule implementation, and decision support. Additionally, we present a detailed classification of approaches used for different applications. Finally, after applying the rigorous review, it was feasible to present the state‐of‐the‐art research, which focuses on designing robust rules, application specific insights, scalable event‐driven architectures and finding the domain's trends with significant challenges and future scope.

Explanatory Denotational Semantics for Complex Event Patterns

Recent years brought popularity and importance of complex event processing (CEP) and associated query languages. CEP systems can be hard to understand. It is often non-trivial to determine streams of events matched by a query, and sometimes we may not notice important edge cases. Hence, the desirability of formal semantics permitting reasoning about resulting complex events and checking if actual matchings agree with our intentions follows. In the article, we introduce a pattern language PatLang with some unique syntactic features related to variable binding. We provide two distinct denotational semantics for PatLang : Minimal semantics , sufficient to describe when patterns match, and tree semantics , which provides detailed information about subpatterns with which the matched events actually match, i.e., information about interpretation of matched events induced by the pattern matching. The tree semantics is unnecessary for verifying correctness of pattern matching execution. However, we show that neither minimal semantics, nor semantics from the prior work suffices to effectively locate errors in patterns with respect to their intended meaning, and that the additional information provided by the tree semantics is crucial for that purpose. We prove that tree semantics can be mapped to minimal semantics. Finally, we provide some practical evaluation.

Geospatial complex event processing in smart city applications

The extensive deployment of the Internet of things (IoT) devices in urban areas enables smart city systems (e.g., urban traffic detection and air pollution monitoring) to use these devices to discover the affected regions by various events, particularly complex ones. A complex event is inferred from patterns of different individual primitive events according to their spatio-temporal information. Complex event processing (CEP) systems facilitate applications to define their reasoning rules based on event processing languages (EPL), similar to SQL. However, current languages and CEP engines do not efficiently support spatial data characteristics. Geospatial CEP is challenging since applications have diverse definitions for complex events (e.g., air quality index), IoT devices have heterogeneous specifications/capabilities, and complex events may happen concurrently. This paper proposes the Geo-Tesla language that enables smart city applications to define spatial attributes in complex event definitions and reasoning rules. Then, we devise the GeoT-Rex CEP engine that implements geospatial operations that compiles Geo-Tesla language and processes the spatial data. The integration of Geo-Tesla and GeoT-Rex enables applications to process complex events with spatial characteristics in their reasoning rules and identify boundaries of the detected complex event (event footprint). The evaluation results from the proof of concept implementation show that our proposed solution outperforms in identifying the footprint of complex events up to 44% in the largest network size. They also show that the detection percentage in the concurrent appearance of complex events is 40% more on average than a close state-of-the-art baseline.

A deep learning-based CEP rule extraction framework for IoT data

With the recent developments in Internet of Things (IoT), the number of sensors that generate raw data with high velocity, variety, and volume is tremendously increased. By employing complex event processing (CEP) systems, valuable information can be extracted from raw data and used for further applications. CEP is a stream processing technology that matches atomic events to complex events via predefined rules mostly created by experts. However, especially in heterogeneous IoT environments, it may become extremely hard to define these rules manually due to complex and dynamic nature of data. Defining rules requires accurate knowledge of relevant events by utilizing temporal dependencies and relations among attributes of events. Therefore, there is a need for an automatic rule extraction system which is capable of generating rules automatically from unlabeled IoT data. This paper proposes a generalized framework for automatic CEP rule extraction with the help of deep learning (DL) methods. The proposed framework contains two main phases which are data labeling and automatic rule extraction phases. We compare several DL methods with each other and regression-based methods to evaluate the proposed framework. In addition, several rule mining methods are employed in the automatic rule extraction phase in a comparative manner. The reconstruction error and prediction success rate of our framework are evaluated using an air quality dataset collected from a smart city application. The results indicate that the proposed framework is able to generate meaningful and accurate rules for unlabeled IoT data.

Complex event processing system for IoT greenhouse

Greenhouse is an important part of facility agriculture and a typical application scenario of modern agricultural technology. The greenhouse environment has the characteristics of nonlinearity, strong coupling, large inertia, and multiple disturbances. There are many environmental factors and it is a typical complex system [7]. In smart greenhouses, control commands are mostly triggered by complex events with multi-dimensional information. In this paper, by building the aggregation structure of complex events in the greenhouse, the technology is applied in the greenhouse as a whole. The core innovations of this paper are as follows: through the analysis of the information transmission process in the greenhouse, combined with the characteristics of the scene, a CEP information structure with predictive modules is formed, which is conducive to the popularization and application of CEP technology in the agricultural field. Pointed out the importance of extreme conditions in the prediction of the greenhouse environment for model evaluation. By improving the loss function in the machine learning algorithm, the prediction performance of a variety of algorithms under this condition has been improved. Applying CEP technology to intelligent greenhouse control scenarios, a set of practical complex event processing systems for greenhouse control has been formed.

State-Aware Load Shedding From Input Event Streams in Complex Event Processing

In complex event processing (CEP), load shedding is performed to maintain a given latency bound during overload situations when there is a limitation on resources. However, shedding load implies degradation in the quality of results (QoR). Therefore, it is crucial to perform load shedding in a way that has the lowest impact on QoR. Researchers, in the CEP domain, propose to drop either events or partial matches (PMs) in overload cases. They assign utilities to events or PMs by considering either the importance of events or the importance of PMs but not both together. In this article, we combine these approaches where we propose to assign a utility to an event by considering both the event importance and the importance of PMs. We propose two load shedding approaches for CEP systems. The first approach drops events from PMs, while the second approach drops events from windows. We adopt a probabilistic model that uses the type and position of an event in a window and the state of a PM to assign a utility to an event. We, also, propose an approach to predict a utility threshold that is used to drop the required amount of events to maintain a given latency bound. By extensive evaluations on two real-world datasets and several representative queries, we show that, in the majority of cases, our load shedding approach outperforms state-of-the-art load shedding approaches, w.r.t. QoR.

Knowledge Graph Driven Approach to Represent Video Streams for Spatiotemporal Event Pattern Matching in Complex Event Processing

Complex Event Processing (CEP) is an event processing paradigm to perform real-time analytics over streaming data and match high-level event patterns. Presently, CEP is limited to process structured data stream. Video streams are complicated due to their unstructured data model and limit CEP systems to perform matching over them. This work introduces a graph-based structure for continuous evolving video streams, which enables the CEP system to query complex video event patterns. We propose the Video Event Knowledge Graph (VEKG), a graph-driven representation of video data. VEKG models video objects as nodes and their relationship interaction as edges over time and space. It creates a semantic knowledge representation of video data derived from the detection of high-level semantic concepts from the video using an ensemble of deep learning models. A CEP-based state optimization — VEKG-Time Aggregated Graph (VEKG-TAG) — is proposed over VEKG representation for faster event detection. VEKG-TAG is a spatiotemporal graph aggregation method that provides a summarized view of the VEKG graph over a given time length. We defined a set of nine event pattern rules for two domains (Activity Recognition and Traffic Management), which act as a query and applied over VEKG graphs to discover complex event patterns. To show the efficacy of our approach, we performed extensive experiments over 801 video clips across 10 datasets. The proposed VEKG approach was compared with other state-of-the-art methods and was able to detect complex event patterns over videos with [Formula: see text]-Score ranging from 0.44 to 0.90. In the given experiments, the optimized VEKG-TAG was able to reduce 99% and 93% of VEKG nodes and edges, respectively, with 5.19[Formula: see text] faster search time, achieving sub-second median latency of 4–20[Formula: see text]ms.

A situation refinement model for complex event processing

Complex Event Processing (CEP) systems aim at processing large flows of events to discover situations of interest (SOI). CEP uses predefined pattern templates to detect occurrences of complex events in an event stream. CEP systems rely on domain experts to define complex patterns rules to recognize SOI. The task of identifying complex patterns faces with several challenges, such as the complexity of writing the pattern rules, and the need to acquire and process background information considering event stream’s real-time constraints. Developing an efficient rule mining algorithm to fine-tune the CEP pattern to recognize SOI requires the tackling of three main obstacles. First, the CEP pattern rules must be inferred by utilizing the user’s preferred context and the history of the event stream. Second, to avoid the issue of pattern complexity, the minimum number of rules must be used in the refinement process. Finally, to respond to emerging situations, the refinement task must be fulfilled in near real-time. In this work, we present a rule mining model to refine the CEP pattern rules by considering these obstacles while providing the ability to adjust the level of refinement to fit the applied scenario. This paper aims to: (1) Review the challenges associated with incorporating domain knowledge in CEP systems to improve awareness of real-world situations; (2) Present a Situation Refinement model to extract the minimal set of rules from external knowledge required to identify Situations Of Interest; (3) Demonstrate the summary update process of the event stream; and (4) Evaluate the derived rules with respect to their coverage and complexity.

Fog-enabled Event Processing Based on IoT Resource Models

Complex Event Processing (CEP) systems extract interest situations from event flows based on event detection patterns. However, local event processing for distributed Internet of Things (IoT) has not been discussed yet. Besides, it is complex or impossible to discover such patterns in some applications of IoT. In this article, we design a complex event service to process event flows based on IoT resource models, which does not depend on existing patterns, and deals with both discrete events and continuous variables. To improve the CEP performance, local IoT resources are used for local event processing, and a lazy exchange method is designed to realize the collaborated event processing between network edges and a data center. Our evaluation shows that our solution is feasible and effective.

Transitions for Increased Flexibility in Fog Computing: A Case Study on Complex Event Processing

Fog computing is envisioned to enable profound applications in the Internet of Things (IoT). A key characteristic of such applications is the need to exchange vital information between distinct IoT devices in the form of event notifications, e. g., traffic conditions when performing traffic monitoring. Complex event processing (CEP) is a powerful paradigm to overcome the information gap from observing primary sensor data by IoT devices to delivering event notifications to the IoT application users. However, to perform CEP in a highly dynamic IoT environment, e. g., involving mobile and heterogeneous devices, require an extremely flexible design of a CEP system to adaptively meet the changing requirements and conditions in which the CEP system is executed. In this article, we show on the use case of CEP, “how to increase flexibility in a fog-cloud computing environment building on a methodology known as mechanism transitions”. In particular, we state and analyze two exemplary IoT use cases to show the potential of mechanism transitions. We identify and discuss possible promising mechanism transitions in the context of CEP. We perform an experimental study for operator placement and show how transitions help to adapt to conflicting performance objectives.

PERCEPTUS: Predictive complex event processing and reasoning for IoT-enabled supply chain

Internet of Things (IoT) is an emerging paradigm that connects various physical sensor devices spread across different locations. IoT-enabled supply chain provides a natural combination to achieve supply chain visibility (SCV) which refers to ability of supply chain partners to collect and analyse distributed supply chain data for the planning and decision support. This data is normally collected and analysed in real-time by a specialized software known as Complex Event Processing (CEP) engines. However, current CEP engines have two well-known limitations. Firstly, current CEP engines are job specific and fail to combine multiple related sensor data streams coming from distributed sources, thereby, supply chain partners are exposed to manage the underlying information heterogeneity. Secondly, these CEP systems do not provide decision support to the supply chain planners when the information about the potential disruptive event is incomplete and/or uncertain. In this paper, a PERCEPTUS framework is proposed to address the above mentioned issues. It, firstly, utilizes semantic annotation process to integrate and annotate events coming from heterogeneous data streams. Secondly, it performs complex event processing to process and correctly interpret annotated complex events. Thirdly, it provides complex event reasoning (by combining logical and probabilistic reasoning) to predict disruption events (such as process failure) under incomplete and/or uncertain information. Finally, the proposed framework is validated using the dataset of a semi-conductor manufacturing process to demonstrate its superiority in terms of accuracy in predicting disruptive events as compared to the baseline approach.

PARS: A parallel model for scaled processing of complex events

The need for fast processing of high volume of event streams has triggered the deployment of parallel processing models and techniques for complex event processing. It is however hard to parallelize stateful operators, making the implementations of distributed complex event processing systems very challenging. A well-defined parallel processing model is a pre-requisite to the implementation of any high performance distributed complex event processing system. Most recent works use partition key or shared memory for parallelizing stateful operators but suffer from imbalanced distribution of keys and slow access time to shared memory. This paper proposes a new parallel model called PARS using three partitioning techniques to allow scalable processing of complex events without using partition key or shared memory. We define PARS formally and use this formalism to prove its soundness and completeness. To present a proof-of-concept and evaluate PARS, we use an event generator to simulate event sources and show significant improvement in system scalability. Compared to other works, we experience 10%–20% higher throughput in our experimental results. Our experiments demonstrate processing speeds of up to 5,200,000 complex event detection per second on a multi-machine cluster.

Managing Measurement and Occurrence Uncertainty in Complex Event Processing Systems

Complex event processing (CEP) is a powerful technology for analyzing streams of real-time events, coming from different sources, and for extracting conclusions from them. In many situations, these events are not free from uncertainty, due to either unreliable data sources and networks, measurement uncertainty, or inability to determine whether an event has actually happened or not. This paper presents a proposal for incorporating and managing different kinds of uncertainty that may happen in both events and rules of the CEP systems. We provide a library that enables the representation and propagation of uncertain values, which can be efficiently integrated with the existing CEP languages and engines to deal with uncertainty, and we show how the treatment of uncertainty can be smoothly added to two of them: Esper and Apache Flink. Five applications coming from various domains serve to evaluate the proposal and to analyze its performance and accuracy. The results show that the overhead introduced by the treatment of uncertainty is not high and good precision and recall are achieved.

Supporting Interval Time Complex Events Processing in an RFID System

Radio Frequency Identification (RFID) is a prevalent wireless identification technology that collects identity information from RFID tags to RFID readers. Complex events processing is an important technique in RFID systems to define the systems status as events and then detect them automatically. Typically the events specifies various of relations of components of a system, where temporal relation is one of such important relations. Existing complex event processing systems specify temporal relations based on the model that events happen instantly. In many applications it is proper to model an event’s occurrence in a time interval, which is called an interval time event. In this paper, we propose a complex event processing approach that can directly specify interval time events and their temporal relations for an RFID system. Thirteen basic temporal relations and several more complex relations are systematically defined for such events. We also propose an approach to build the automaton for the specified events, which can be executed for detecting the events. The evaluation results show that the proposed approach outperforms existing approaches.

A Universal Complex Event Processing Mechanism Based on Edge Computing for Internet of Things Real-Time Monitoring

The tremendous number of sensors and smart objects deployed in the Internet of Things (IoT) pose a huge potential for the IoT real-time monitoring applications to detect and react to the real world. The insufficient capacity of the IoT data real-time processing has hampered the growth of the IoT real-time monitoring applications. We focus on two issues of the IoT data real-time processing: 1) how to efficiently transform a large number of raw sensing data into meaningful complex event, and 2) how to adapt to the complexity and changeability of monitoring business logic. This paper proposes a universal complex event processing (CEP) mechanism for the IoT real-time monitoring. We propose a formalized hierarchical complex event model including raw event, simple event, and complex event, which reduces the complexity of event modeling. The model supports complex time and space semantics to define flexible complex events by a programming way. Based on this model, we propose a CEP system architecture, in which the system is deployed on the network edge between sensing devices in terminal and applications in the cloud. The complex event definition can be mapped to the CEP rule logic script to detect the potential abnormal event timely. The proposed CEP mechanism is universal and suitable to any heterogeneous sensing devices and CEP engine. We demonstrate the efficacy of the mechanism with two application case studies that highlight our proposed complex event model and evaluate the performance improvement with experiments.

An Adaptive Parallel Processing Strategy for Complex Event Processing Systems over Data Streams in Wireless Sensor Networks.

Efficient matching of incoming events of data streams to persistent queries is fundamental to event stream processing systems in wireless sensor networks. These applications require dealing with high volume and continuous data streams with fast processing time on distributed complex event processing (CEP) systems. Therefore, a well-managed parallel processing technique is needed for improving the performance of the system. However, the specific properties of pattern operators in the CEP systems increase the difficulties of the parallel processing problem. To address these issues, a parallelization model and an adaptive parallel processing strategy are proposed for the complex event processing by introducing a histogram and utilizing the probability and queue theory. The proposed strategy can estimate the optimal event splitting policy, which can suit the most recent workload conditions such that the selected policy has the least expected waiting time for further processing of the arriving events. The proposed strategy can keep the CEP system running fast under the variation of the time window sizes of operators and the input rates of streams. Finally, the utility of our work is demonstrated through the experiments on the StreamBase system.

Research on Complex Event Detection over Streams Supporting Interval Temporal Logic and Regular Expression Pattern Matching

To improve the detective ability of complex event processing (CEP), we combine interval temporal logic (ITL) and regular expression in pattern description of event streams constraint. We propose different timed automata to support varied ITL. The pattern described by regular expression improves the simplicity of the pattern description. Meanwhile, we have implemented above technique in a CEP system to test performance. The experimental evaluation shows that the pattern can be efficiently and accurately matched.

Top-k 질의 처리를 위한 오픈소스 복합 이벤트 처리 시스템의 확장

이벤트 스트림에 대한 Top-k 질의는 현재까지 유입된 이벤트 중 주어진 속성값이 큰 상위 k의 이벤트를 내어주는 것이다. 본 논문에서는 오픈소스 이벤트스트림 처리기인 Esper 엔진이 Top-k 질의를 효율적으로 지원할 수 있도록 확장하는 방법을 소개한다. 이를 위해 먼저 Esper 질의 언어의 문법을 Top-k 질의 지원을 위해 확장하였고, 상위 k개의 이벤트만 정렬하여 사용하는 효과적인 처리기를 제작하였다. 제작한 처리기와 Esper와의 결합을 위해서 추출 값 뷰(Derived View)를 확장하여 Top-k 기능을 구현하였다. 또한 제안한 방법은 기존의 Esper에서 제공되는 질의문 만을 통해서 작성한 질의에 비해 월등한 성능을 보이는 것을 실험으로 보였다.

Formalizing Complex Event Processing Systems in Maude

Complex event processing (CEP) is a cutting-edge technology for analyzing and correlating streams of information about events that happen in a system, and deriving conclusions from them. CEP permits defining complex events based on the events produced by the incoming sources, to identify complex meaningful circumstances and to respond to them as quickly as possible. Such event types and patterns are defined using event processing languages. However, as the complexity of CEP programs grows, they become difficult to understand and to prove correct. This paper proposes a formal framework for the specification of CEP applications, using rewriting logic and Maude, to allow developers to formally analyze and prove properties of their CEP programs. Several case studies are presented to illustrate the approach, as well as a discussion on the benefits of using Maude and its toolkit for modeling and analyzing CEP systems.