Complex Event Processing Rules Research Articles

Learning Ship Activity Patterns in Maritime Data Streams: Enhancing CEP Rule Learning by Temporal and Spatial Relations and Domain-Specific Functions

Maritime surveillance systems are of particular importance for the smooth and safe operation of maritime traffic. Such systems must efficiently analyze the continuous stream of incoming ship movement data to provide an up-to-date picture of the situation at sea. Complex Event Processing (CEP) is a software technology dedicated to the analysis of data streams in real time and seems to be promising for maritime surveillance as well. CEP is based on rules that usually have to be formulated manually by domain experts. Recently, several approaches of learning CEP rules have been proposed. However, all these approaches are limited to standard CEP rule languages which are not well suited for describing maritime situations. In this paper, we extend Bat4CEP, an innovative bat-inspired approach to automatic CEP rule learning, to meet the specific needs of the maritime domain. In particular, we extend the rule language of Bat4CEP to include temporal-spatial operators and domain-specific functions. This extended rule language makes it possible to express complex maritime facts in a simple and understandable form. Furthermore, it is shown how the proposed rule language extensions can be integrated into the Bat4CEP learning approach. The effectiveness of the approach is demonstrated through extensive experiments with real maritime data streams.

An automatic complex event processing rules generation system for the recognition of real-time IoT attack patterns

The Internet of Things (IoT) has grown rapidly to become the core of many areas of application, leading to the integration of sensors, with IoT devices. However, the number of attacks against these types of devices has grown as fast as the paradigm itself. Certain inherent characteristics of the paradigm, as well as the limited computational capabilities of the devices involved, make it difficult to deploy security measures. This is why it is necessary to design, implement and study new solutions in the field of cybersecurity. In this paper, we propose an architecture that is capable of generating Complex Event Processing (CEP) rules automatically by integrating them with machine learning technologies. While the former is used to automatically detect attack patterns in real time, the latter, through the use of the Principal Component Analysis (PCA) algorithm, allows the characterization of events and the recognition of anomalies. This combination makes it possible to achieve efficient CEP rules at the computational level, with the results showing that the CEP rules obtained using our approach substantially improve upon the performance of the standard CEP rules, which are rules that are not generated by our proposal but can be defined independently by an expert in the field. Our proposal has achieved an F1-score of 0.98 on average, a 76 percent improvement in throughput over standard CEP rules, and a reduction in the network overhead of 86 percent over standard simple events, which are the simple events that are generated when our proposal is not used.

An automatic unsupervised complex event processing rules generation architecture for real-time IoT attacks detection

AbstractIn recent years, the Internet of Things (IoT) has grown rapidly, as has the number of attacks against it. Certain limitations of the paradigm, such as reduced processing capacity and limited main and secondary memory, make it necessary to develop new methods for detecting attacks in real time as it is difficulty to adapt as has the techniques used in other paradigms. In this paper, we propose an architecture capable of generating complex event processing (CEP) rules for real-time attack detection in an automatic and completely unsupervised manner. To this end, CEP technology, which makes it possible to analyze and correlate a large amount of data in real time and can be deployed in IoT environments, is integrated with principal component analysis (PCA), Gaussian mixture models (GMM) and the Mahalanobis distance. This architecture has been tested in two different experiments that simulate real attack scenarios in an IoT network. The results show that the rules generated achieved an F1 score of .9890 in detecting six different IoT attacks in real time.

An Auto-Extraction Framework for CEP Rules Based on the Two-Layer LSTM Attention Mechanism: A Case Study on City Air Pollution Forecasting

Energy is at the center of human society and drives the technologies and overall human well-being. Today, artificial intelligence (AI) technologies are widely used for system modeling, prediction, control, and optimization in the energy sector. The internet of things (IoT) is the core of the third wave of the information industry revolution and AI. In the energy sector, tens of billions of IoT appliances are linked to the Internet, and these appliances generate massive amounts of data every day. Extracting useful information from the massive amount of data will be a very meaningful thing. Complex event processing (CEP) is a stream-based technique that can extract beneficial information from real-time data through pre-establishing pattern rules. The formulation of pattern rules requires strong domain expertise. Therefore, at present, the pattern rules of CEP still need to be manually formulated by domain experts. However, in the face of complex, massive amounts of IoT data, manually setting rules will be a very difficult task. To address the issue, this paper proposes a CEP rule auto-extraction framework by combining deep learning methods with data mining algorithms. The framework can automatically extract pattern rules from unlabeled air pollution data. The deep learning model we presented is a two-layer LSTM (long short-term memory) with an attention mechanism. The framework has two phases: in the first phase, the anomalous data is filtered out and labeled from the IoT data through the deep learning model we proposed, and then the pattern rules are mined from the labeled data through the decision tree data mining algorithm in the second phase. We compare other deep learning models to evaluate the feasibility of the framework. In addition, in the rule extraction stage, we use a decision tree data mining algorithm, which can achieve high accuracy. Experiments have shown that the framework we proposed can effectively extract meaningful and accurate CEP rules. The research work in this paper will help support the advancement of the sector of air pollution prediction, assist in the establishment of air pollution regulatory strategies, and further contribute to the development of a green energy structure.

Bat4CEP: a bat algorithm for mining of complex event processing rules

Complex Event Processing (CEP) is a modern software technology for the dynamic analysis of continuous data streams. CEP is able of searching extremely large data streams in real time for the presence of event patterns. So far, specifying event patterns of CEP rules is still a manual task based on the expertise of domain experts. This paper presents a novel bat-inspired swarm algorithm for automatically mining CEP rule patterns that express the relevant causal and temporal relations hidden in data streams. The basic suitability and performance of the approach is proven by extensive evaluation with both synthetically generated data and real data from the traffic domain.

Rule‐based preprocessing for data stream mining using complex event processing

AbstractData preprocessing is known to be essential to produce accurate data from which mining methods are able to extract valuable knowledge. When data constantly arrives from one or more sources, preprocessing techniques need to be adapted to efficiently handle these data streams. To help domain experts to define and execute preprocessing tasks for data streams, this paper proposes the use of active rule‐based systems and, more specifically, complex event processing (CEP) languages and engines. The main contribution of our approach is the formulation of preprocessing procedures as event detection rules, expressed in an SQL‐like language, that provide domain experts a simple way to manipulate temporal data. This idea is materialized into a publicly available solution that integrates a CEP engine with a library for online data mining. To evaluate our approach, we present three practical scenarios in which CEP rules preprocess data streams with the aim of adding temporal information, transforming features and handling missing values. Experiments show how CEP rules provide an effective language to express preprocessing tasks in a modular and high‐level manner, without significant time and memory overheads. The resulting data streams do not only help improving the predictive accuracy of classification algorithms, but also allow reducing the complexity of the decision models and the time needed for learning in some cases.

A deep learning-based CEP rule extraction framework for IoT data

With the recent developments in Internet of Things (IoT), the number of sensors that generate raw data with high velocity, variety, and volume is tremendously increased. By employing complex event processing (CEP) systems, valuable information can be extracted from raw data and used for further applications. CEP is a stream processing technology that matches atomic events to complex events via predefined rules mostly created by experts. However, especially in heterogeneous IoT environments, it may become extremely hard to define these rules manually due to complex and dynamic nature of data. Defining rules requires accurate knowledge of relevant events by utilizing temporal dependencies and relations among attributes of events. Therefore, there is a need for an automatic rule extraction system which is capable of generating rules automatically from unlabeled IoT data. This paper proposes a generalized framework for automatic CEP rule extraction with the help of deep learning (DL) methods. The proposed framework contains two main phases which are data labeling and automatic rule extraction phases. We compare several DL methods with each other and regression-based methods to evaluate the proposed framework. In addition, several rule mining methods are employed in the automatic rule extraction phase in a comparative manner. The reconstruction error and prediction success rate of our framework are evaluated using an air quality dataset collected from a smart city application. The results indicate that the proposed framework is able to generate meaningful and accurate rules for unlabeled IoT data.

Learning of complex event processing rules with genetic programming

Complex Event Processing (CEP) is an established software technology to extract relevant information from massive data streams. Currently, domain experts have to determine manually CEP rules that define a situation of interest. However, often CEP rules cannot be formulated by experts, because the relevant interdependencies and relations between the data are not explicitly known, but inherently hidden in the data streams. To cope with this problem, we present a new learning approach for CEP rules, which is based on Genetic Programming. We discuss in detail the different building blocks of Genetic Programming and how to adjust them to CEP rule learning. Extensive evaluations with synthetic and real world data demonstrate the high potential of the approach and give some hints about the choice of suitable process parameters.

A Universal Complex Event Processing Mechanism Based on Edge Computing for Internet of Things Real-Time Monitoring

The tremendous number of sensors and smart objects deployed in the Internet of Things (IoT) pose a huge potential for the IoT real-time monitoring applications to detect and react to the real world. The insufficient capacity of the IoT data real-time processing has hampered the growth of the IoT real-time monitoring applications. We focus on two issues of the IoT data real-time processing: 1) how to efficiently transform a large number of raw sensing data into meaningful complex event, and 2) how to adapt to the complexity and changeability of monitoring business logic. This paper proposes a universal complex event processing (CEP) mechanism for the IoT real-time monitoring. We propose a formalized hierarchical complex event model including raw event, simple event, and complex event, which reduces the complexity of event modeling. The model supports complex time and space semantics to define flexible complex events by a programming way. Based on this model, we propose a CEP system architecture, in which the system is deployed on the network edge between sensing devices in terminal and applications in the cloud. The complex event definition can be mapped to the CEP rule logic script to detect the potential abnormal event timely. The proposed CEP mechanism is universal and suitable to any heterogeneous sensing devices and CEP engine. We demonstrate the efficacy of the mechanism with two application case studies that highlight our proposed complex event model and evaluate the performance improvement with experiments.

Online Monitoring of Manufacturing Process Based on autoCEP

Complex Event Processing (CEP), which can identify patterns of interest from a large number of continuous data steam, is becoming more and more popular in manufacturing process monitoring. CEP rules are specified manually by domain expert, which is a limiting factor for its application in manufacturing enterprises. How to analysis historical data and automatically generate CEP rules is becoming a challenge research. This paper proposed a model of autoCEP for online monitoring in product manufacturing, which can automatically generate CEP rules based on association rules mining in key processes. First, the key quality factors in manufacturing process were extracted by grey entropy correlation analysis. Then, association rules mining method based on product process constraints was used to find the association rules between key factors and product quality. At last, the extracted rules are algorithmically transformed into CEP rules. The experimental results show the effectiveness and practicability of the proposed method.

Towards anticipate detection of complex event processing rules with probabilistic modelling

Nowadays, Big Data implies not only the need of processing high volume of data, but also do it in a timely manner. In this scope, the Complex Event Processing (CEP) paradigm has arisen as a prominent real-time rule-based solution. Due to its reactive nature, a CEP system might suffer from slight delays in the activation of its rules that could not be desirable in certain environments. As a result, the present work introduces a novel mechanism that intends to anticipate the activation of event-based rules and, thus, come up with even faster CEP systems. This is achieved by means of a probabilistic modelling of each rule’s precondition. Finally, the proposal includes a preliminary evaluation so as to show its suitability.

A Validation System for the Complex Event Processing Directives of the ATLAS Shifter Assistant Tool

Complex Event Processing (CEP) is a methodology that combines data from many sources in order to identify events or patterns that need particular attention. It has gained a lot of momentum in the computing world in the past few years and is used in ATLAS to continuously monitor the behaviour of the data acquisition system, to trigger corrective actions and to guide the experiment's operators. This technology is very powerful, if experts regularly insert and update their knowledge about the system's behaviour into the CEP engine. Nevertheless, writing or modifying CEP rules is not trivial since the used programming paradigm is quite different with respect to what developers are normally familiar with. In order to help experts verify that the rules work as expected, we have thus developed a complete testing and validation environment. This system consists of three main parts: the first is the data reader from existing storage of all relevant data streams that are produced during data taking, the second is a playback tool that allows to re-inject data of specific data taking sessions from the past into the CEP engine, and the third is a reporting tool that shows the output that the rules loaded into the engine would have produced in the live system. In this paper we describe the design and implementation of this validation system, highlight its strengths and shortcomings and indicate how such a system could be reused in similar projects.