Complex Event Processing Technology Research Articles

Leveraging complex event processing for monitoring and automatically detecting anomalies in Ethereum-based blockchain networks

Blockchain is a relatively recent technology that provides immutability, traceability and transparency of information, thus building trust in the digital society. Blockchain networks generate a large amount of logs which capture and describe data flowing through the network in the form of transactions, blocks and events. Monitoring these blockchain data from the off-chain world is needed to detect anomalies with the aim of mitigating the risks that may arise as a result of using blockchain technology. However, the real-time monitoring of these logs by off-chain systems has become a challenge from the beginning of 2018 when the blockchain networks reached a high number of daily transactions. In this paper, we propose a portable, maintainable and easily configurable architecture integrating blockchain and complex event processing technologies that allows for both the real-time monitoring of logs generated in Ethereum Virtual Machine (EVM)-compatible blockchain networks and the automatic detection of anomalies in these networks by matching event patterns. This architecture was tested by using vast amounts of blockchain data already publicly registered in Ethereum and Polygon networks. The results demonstrate that the proposed architecture is able to automatically detect anomalies which occur in different blockchain networks, making analytics of blockchain data possible by off-chain systems.

Taxonomy and software architecture for real-time context-aware collaborative smart environments

The widespread of Internet of Things (IoT) and the price reduction and ubiquity of telecommunications has led to the emergence of smart environments where devices are becoming increasingly smarter and everything is connected and from which society aims to benefit. The data obtained from IoT is rapidly processed in various domains for the achievement of smart cities and societies. However, in many cases, applications are not contextualized by using data from outside the domain but are only contextualized using data from the domain itself, missing the opportunity for further contextualization. The lack of common criteria for the integration of data from different application domains is one of the main reasons that significantly hinders the integration of third-party data into real-time processing and decision-making systems and thus, the context awareness of developed applications. Although the use of several taxonomies and ontologies for context awareness in various application domains have been proposed, in many cases they are highly domain specific and/or difficult to integrate with other systems, which makes it challenging to facilitate data sharing between different systems and their processing to achieve enhanced context awareness. We aim to contribute to the addressing of these limitations through a reusable and extensible multi-domain taxonomy targeted to collaborative IoT and smart environments, which is also automatically integrated into a software architecture with real-time complex event processing technologies. The proposed solution has been illustrated through a case study and performance tests have been carried out in different computing capacity scenarios, showing its feasibility and usefulness.

Evaluating the integration of Esper complex event processing engine and message brokers.

The great advance and affordability of technologies, communications and sensor technology has led to the generation of large amounts of data in the field of the Internet of Things and smart environments, as well as a great demand for smart applications and services adapted to the specific needs of each individual. This has entailed the need for systems capable of receiving, routing and processing large amounts of data to detect situations of interest with low latency, but despite the many existing works in recent years, studying highly scalable and low latency data processing systems is still necessary. In this area, the efficiency of complex event processing (CEP) technology is of particular significance and has been used in a variety of application scenarios. However, in most of these scenarios there is no performance evaluation to show how the system performs under various loads and therefore the developer is challenged to develop such CEP-based systems in new scenarios without knowing how the system will be able to handle different input data rates and address scalability and fault tolerance. This article aims to fill this gap by providing an evaluation of the various versions of one of the most reputable CEP engines-Esper CEP, as well as its integration with two renowned messaging brokers for data ingestion-RabbitMQ and Apache Kafka. For this purpose, we defined a benchmark with a series of event patterns with some of the most representative operators of the Esper CEP engine and we performed a series of tests with an increasing rate of input data to the system. We did this for three alternative software architectures: integrating open-source Esper and RabbitMQ, integrating one instance of Esper enterprise edition with Apache Kafka, and integrating two distributed instances of Esper enterprise edition with Apache Kafka. We measured the usage of CPU, RAM memory, latency and throughput time, looking for the data input rate with which the system overloads for each event pattern and we compared the results of the three proposed architectures. The results have shown a very low CPU consumption for all implementation options and input data rates; a balanced memory usage, quite similar among the three architectures, up to an input rate of 10,000 or 15,000 events per second, depending on the architecture and event pattern, and a quite efficient response time up to 10,000 or 15,000 events per second, depending on the architecture and event pattern. Based on a more exhaustive analysis of results, we have concluded that the different options offered by Esper for CEP provide very efficient solutions for real-time data processing, although each with its limitations in terms of brokers to be used for data integration, scalability, and fault tolerance; a number of suggestions have been drawn out for the developer to take as a basis for choosing which CEP engine and which messaging broker to use for the implementation depending on the of the system in question.

CEPEDALoCo: An event-driven architecture for integrating complex event processing and blockchain through low-code

Internet of Things (IoT) is made up of millions of devices generating large amounts of heterogeneous data from multiple sources. These devices can be from multiple manufacturers which makes their use in terms of data acquisition, processing, analysis and actions on these data challenging. Additionally, all these data must be analyzed and correlated in real time for the early detection of situations of interest (complex events) and subsequent decision making. These complex events must be able to automatically trigger decisions and be stored in a secure, immutable and accessible way. In this context, Event-Driven Applications (EDAs) are a solution to meet these needs, however, developing such applications requires vast knowledge in certain technologies. To address these challenges, an EDA is proposed in this paper to integrate Complex Event Processing (CEP) and blockchain through the low-code paradigm. This proposal allows for the development of EDAs in a user-friendly way. These applications make it possible to integrate IoT devices from multiple manufacturers and with different data formats together with CEP technology for complex event detection and blockchain for secure, immutable and accessible event storage. To demonstrate the feasibility, the architecture was applied and evaluated in a case study related to measuring and acting on air quality using IoT devices that measure different pollutants and factors such as temperature, humidity and wind. The results show that the graphically designed EDAs facilitate the real-time analysis of the collected IoT data via a CEP engine, whose outcome is transparently and automatically registered in a blockchain network.

An automatic unsupervised complex event processing rules generation architecture for real-time IoT attacks detection

In recent years, the Internet of Things (IoT) has grown rapidly, as has the number of attacks against it. Certain limitations of the paradigm, such as reduced processing capacity and limited main and secondary memory, make it necessary to develop new methods for detecting attacks in real time as it is difficulty to adapt as has the techniques used in other paradigms. In this paper, we propose an architecture capable of generating complex event processing (CEP) rules for real-time attack detection in an automatic and completely unsupervised manner. To this end, CEP technology, which makes it possible to analyze and correlate a large amount of data in real time and can be deployed in IoT environments, is integrated with principal component analysis (PCA), Gaussian mixture models (GMM) and the Mahalanobis distance. This architecture has been tested in two different experiments that simulate real attack scenarios in an IoT network. The results show that the rules generated achieved an F1 score of .9890 in detecting six different IoT attacks in real time.

FUME: An air quality decision support system for cities based on CEP technology and fuzzy logic

Air pollution has become one of the most important problems in urban areas, and governments are applying regulations in an attempt to fulfill the recommendations of Air Quality (AQ) standards to reduce the pollution. In this paper, we present FUME, a decision support system to process heterogeneous and real-time data to propose daily recommendations following an action protocol based on AQ standards. This approach considers past, current and future environmental situations (AQ and atmospheric stability). FUME is implemented by combining Fuzzy Logic (FL) and Complex Event Processing (CEP) technology. In particular, we propose a Fuzzy Inference System (FIS) to improve the decision-making process by recommending the actions for four different sources of pollution: traffic, industry, domestic and agriculture. The FUME approach is applied to a specific case study: the city of Puertollano (Ciudad Real, Spain), where the pollution levels of PM10 are, on numerous occasions, above the World Health Organization (WHO) recommendations.

Beholder – A CEP-based intrusion detection and prevention systems for IoT environments

IoT devices are vulnerable to security threats. The connectivity among such devices presents opportunities for cybercriminals to perform different types of attacks. In IoT, application layer protocols play an important role. They form the basis of communications between applications and services. Security is an optional design aspect in application layer protocols, leading developers to neglect these security settings, or others may lack security mechanisms in their standard configuration. This paper introduces Beholder, an Intrusion Detection and Prevention System (IDPS) to prevent attacks that exploit the vulnerabilities in the MQTT (Message Queue Telemetry Transport) and CoAP (Constrained Application Protocol) protocols. Most IoT systems adopt MQTT or CoAP. Beholder exploits the Complex Event Processing (CEP) technology to analyze data streams and detect attacks on IoT applications that use the MQTT and CoAP protocols. Beholder is the first IDS that relies on CEP technology to detect attacks on MQTT and CoAP protocols in IoT environments to the best of our knowledge. This work introduces a real scenario for the evaluation process, performing attacks on a smart home. Initially, we validated the quality of our IDS, comparing it against the well-known Snort IDS. Since Snort can only handle attacks on TCP, we restricted the experiment to attacks on this protocol. Results demonstrated that Snort achieved three 9s of precision, while Beholder reached a precision of four 9s. The experiment to evaluate Beholder’s precision for the MQTT and CoAP protocols revealed that Beholder obtained 100% precision for High QoS Flood (MQTT) and the Error Flood(CoAP).

Dynamic Performance Analysis of STEP System in Internet of Vehicles Based on Queuing Theory.

The Internet of vehicles (IoV) is an important research area of the intelligent transportation systems using Internet of things theory. The complex event processing technology is a basic issue for processing the data stream in IoV. In recent years, many researchers process the temporal and spatial data flow by complex event processing technology. Spatial Temporal Event Processing (STEP) is a complex event query language focusing on the temporal and spatial data flow in Internet of vehicles. There are four processing models of the event stream processing system based on the complex event query language: finite automata model, matching tree model, directed acyclic graph model, and Petri net model. In addition, the worst-case response time of the event stream processing system is an important indicator of evaluating the performance of the system. Firstly, this paper proposed a core algorithm of the temporal and spatial event stream processing program based on STEP by Petri net model. Secondly, we proposed a novel method to estimate the worst-case response time of the event stream processing system, which is based on stochastic Petri net and queuing theory. Finally, through the simulation experiment based on queuing theory, this paper proves that the data stream processing system based on STEP has good dynamic performance in processing the spatiotemporal data stream in Internet of vehicles.

Online correlation for unlabeled process events: A flexible CEP-based approach

Process mining is a sub-field of data mining that focuses on analyzing timestamped and partially ordered data. This type of data is commonly called event logs. Each event is required to have at least three attributes: case ID, task ID/name, and timestamp to apply process mining techniques. Thus, any missing information need to be supplied first. Traditionally, events collected from different sources are manually correlated. While this might be acceptable in an offline setting, this is infeasible in an online setting. Recently, several use cases have emerged that call for applying process mining in an online setting. In such scenarios, a stream of high-speed and high-volume events continuously flow, e.g. IoT applications, with stringent latency requirements to have insights about the ongoing process. Thus, event correlation must be automated and occur as the data is being received.We introduce an approach that correlates unlabeled events received on a stream. Given a set of start activities, our approach correlates unlabeled events to a case identifier. Our approach is probabilistic. That implies a single uncorrelated event can be assigned to zero or more case identifiers with different probabilities. Moreover, our approach is flexible. That is, the user can supply domain knowledge in the form of constraints that reduce the correlation space. This knowledge can be supplied while the application is running. We realize our approach using complex event processing (CEP) technologies. We implemented a prototype on top of Esper, a state of the art industrial CEP engine. We compare our approach to baseline approaches. The experimental evaluation shows that our approach outperforms the throughput and latency of the baseline approaches. It also shows that using real-life logs, the accuracy of our approach can compete with the baseline approaches.

GreenITS: a proposal to compute low-pollution routes

A way to reduce carbon emissions in cities is through movement by bicycle or on foot. However, it sometimes means to pass through high-pollution zones and consequently breath low quality air. We then propose a green Intelligent Transportation System (ITS) for zero-emission mobility users, providing users with low-pollution routes to avoid the high-pollution zones. This proposal uses ITS to promote the use of alternative transportation to classical motor vehicles to reduce carbon emissions. This is based on Complex Event Processing (CEP) technology to gather and process real-time data, a Decision Support System designed as a Fuzzy Inference System (FIS) to make decisions about recommended transit zones, taking also into account the user experience level and specific weather data, and Colored Petri Nets (CPN) as a tool to compute the routes. This is therefore an all-in-one solution to provide green routes, with the benefits of each one of the technologies used.

DARLING

Complex event processing (CEP) is widely employed to detect user-defined combinations, or patterns, of events in massive streams of incoming data. Numerous applications such as healthcare, fraud detection, and more, use CEP technologies to capture critical alerts, threats, or vital notifications. This requires that the technology meet real-time detection constraints. Multiple optimization techniques have been developed to minimize the processing time for CEP, including parallelization techniques, pattern rewriting, and more. However, these techniques may not suffice or may not be applicable when an unpredictable peak in the input event stream exceeds the system capacity. In such cases, one immediate possible solution is to drop some of the load in a technique known as load shedding. We present a novel load shedding mechanism for real-time complex event processing. Our approach uses statistics that are gathered to detect overload. The solution makes data-driven load shedding decisions to drop the less important events such that we preserve a given latency bound while minimizing the degradation in the quality of results. An extensive experimental evaluation on a broad set of real-life patterns and datasets demonstrates the superiority of our approach over the state-of-the-art techniques.

Complex Event Processing Methods for Greenhouse Control

In the greenhouse environment, the application of complex event processing (CEP) technology can effectively tackle the problem of recognition of the complex patterns that appeared in greenhouse conditions. In the existing research, few scholars have proposed a scheme to integrate complicated scenes within the greenhouse environment with high efficiency, convenience, and low coupling. Therefore, in order to solve the problem of hard recognition and fusion of complex patterns in the greenhouse environment, based on the characteristics of the greenhouse, this paper proposes a complex event processing method for greenhouse control. Our method has high applicability and high expansibility, including 13 types of event processing agents and 21 types of typical events involved in greenhouse automatic control. This method has the advantages of low information coupling and multi-domain integration, which can be directly used by agricultural experts and related workers and is of great significance to promote the extensive application of CEP technology in the greenhouse field. Our experiment successfully realized a greenhouse intelligent control system based on CEP technology is successfully realized in our experiment. The experimental statistics shows that the structure of the control system was accessible and effective.

MEdit4CEP-SP: A model-driven solution to improve decision-making through user-friendly management and real-time processing of heterogeneous data streams

Organisations today are constantly consuming and processing huge amounts of data. Such datasets are often heterogeneous, making it difficult to work with them quickly and easily due to their format constraints or their disparate data structures. Therefore, being able to efficiently and intuitively work with such data to analyse them in real time to detect situations of interest as quickly as possible is a great competitive advantage for companies. Existing approaches have tried to address this issue by providing users with analytics or modelling tools in an isolated way, but not combining them as a one-in-all solution. In order to fill this gap, we present MEdit4CEP-SP, a model-driven system that integrates Stream Processing (SP) and Complex Event Processing (CEP) technologies for consuming, processing and analysing heterogeneous data in real time. It provides domain experts with a graphical editor that allows them to infer and define heterogeneous data domains, while also modelling, in a user-friendly way, the situations of interest to be detected in such domains. These graphical definitions are then automatically transformed into code, which is deployed in the processing system at runtime. The alerts detected by the system, in real-time, allow users to react as quickly as possible, thus improving the decision-making process. Additionally, MEdit4CEP-SP provides persistence, storing these definitions in a NoSQL database to permit their reuse by other instances of the system. Further benefits of this system are evaluated and compared with other existing approaches in this paper.

Integrating complex event processing and machine learning: An intelligent architecture for detecting IoT security attacks

The Internet of Things (IoT) is growing globally at a fast pace: people now find themselves surrounded by a variety of IoT devices such as smartphones and wearables in their everyday lives. Additionally, smart environments, such as smart healthcare systems, smart industries and smart cities, benefit from sensors and actuators interconnected through the IoT. However, the increase in IoT devices has brought with it the challenge of promptly detecting and combating the cybersecurity attacks and threats that target them, including malware, privacy breaches and denial of service attacks, among others. To tackle this challenge, this paper proposes an intelligent architecture that integrates Complex Event Processing (CEP) technology and the Machine Learning (ML) paradigm in order to detect different types of IoT security attacks in real time. In particular, such an architecture is capable of easily managing event patterns whose conditions depend on values obtained by ML algorithms. Additionally, a model-driven graphical tool for security attack pattern definition and automatic code generation is provided, hiding all the complexity derived from implementation details from domain experts. The proposed architecture has been applied in the case of a healthcare IoT network to validate its ability to detect attacks made by malicious devices. The results obtained demonstrate that this architecture satisfactorily fulfils its objectives.

QuantCloud: Enabling Big Data Complex Event Processing for Quantitative Finance Through a Data-Driven Execution

Quantitative Finance (QF) utilizes increasingly sophisticated mathematic models and advanced computer techniques to predict the movement of global markets, and price the derivatives and other assets. Being able to react quickly and intelligently to fast-changing markets is a decisive success factor for trading companies. To date, the rise of QF requires an integrated toolchain of enabling technologies to carry out complex event processing on the explosive growth and diversified forms of market metadata, in pursuit of a microsecond latency on an Exabyte-level dataset. Inspired by this, we present a data-driven execution paradigm that untangles the dependencies of complex processing events and integrate the paradigm with a big data infrastructure that streams time series data. This integrated platform is termed as the QuantCloud platform. Essentially, QuantCloud executes the complex event processing in a data-driven mode and manages large amounts of diversified market data in a data-parallel mode. To show its practicability and performance, we develop a prototype and benchmark by applying real-world QF research models on the New York Stock Exchange (NYSE) data. Using this prototype, we demonstrate this platform with an application to: (i) data cleaning and aggregating (including the computing of logarithmic returns from tick data and the finding the medians of grouped data) and (ii) data modeling: the autoregressive-moving average (ARMA) model. The performance results show that (a) this platform obtains a high throughput (usually in the order of millions of tick messages per second) and a sub-microsecond latency; (b) it fully executes data-dependent tasks through a data-driven execution; and (c) it implements a modular design approach for rapidly developing these data-crunching methods and QF research models. This platform resulting from an aggregated effort of the data-driven execution and big data infrastructure, offers the financial engineers with new insights and enhanced capabilities for effective and efficient incorporation of big data complex event processing technologies in their workflow.

Closed loop management system algorithms of acquisition operations and maintenance based on alternation early warning

In order to construct the closed-loop management system of abnormal early warning, the decision tree algorithm, survival analysis algorithm and logistic regression algorithm are used synthetically. The improved logistic regression algorithm proposed in this study is used to establish the abnormal e arly warning model, identify the abnormal tendency in advance, and construct the active monitoring and closed-loop operation and maintenance management system to provide the key technical support. The results show that this method improves the real-time processing of data acquisition, reduces the impact on data acquisition, improves the efficiency of operation and maintenance, and carries out timely and accurate early warning. This method proposes active monitoring mechanism based on complex event processing and fast fault location technology based on expert database. From this point of view, testing the key technical indicators of the system proves that the system can effectively promote the improvement of business operation and maintenance efficiency, quality and management level.

An Architecture with Mixed Complex Event Processing Technology For CPS

Cyber-Physical System (CPS) is an emerging intelligent systems, which is considered as a combination of computers and physical world. Events are usually as a natural way for interactions and communications, which occur in the physical world and can be responded in the cyber world. Additionally, using complex event processing (CEP) technology, some complex events can be processed that are better suited for complicated and intelligent CPS. At first, a component-model framework is proposed for modeling CPS architecture based on event-driven pattern operational semantics, which provides a tool to understand the transition based on event-driven pattern in CPS. In order to improve the immense workload on centralized CEP, a mixed master-slaves CEP is proposed. Meanwhile, an architecture of CPS with master-slaves CEP is constructed and its execution mechanism is introduced in detail.

Research on urban anti-terrorism intelligence perception system from the perspective of Internet of things application

Objective/significance Based on the urgent needs of the status quo of urban anti-terrorism, combining the present situation of intelligent city construction, we carry out the research on the key problems such as the collection and analysis of counter-terrorism information for urban environment. This paper puts forward a kind of city anti-terrorism information perception system based on the Internet of things, and realizes to improve the level of surveillance and early warning of terrorist activities facing the urban environment. Method/Process The system collects and merges the heterogeneous information of terrorism, which is based on the information perception layer, the intelligence identification layer and the intelligence inference layer, which is based on the city's focal area. To filter suspicious information by constructing a semantic database of terrorist action signals Combined with complex event processing and machine learning technology, real-time monitoring and mining prediction of mass event flow, using the comparison between historical data of terrorist attacks and real-time data, increased the area of counter-terrorism intelligence. Results/Conclusions We hope to provide a better model and an exploratory study on the construction of the counter-terrorism early warning for the urban environment.

Street Lamp Fault Diagnosis System Based on Extreme Learning Machine

In view of the construction of urban lighting system needs a lot of manpower deployment, especially for its fault diagnosis problem management. This paper proposes a fault model detection and diagnosis subsystem based on the extreme learning machine for street lamps system. The subsystem is part of the event rule response system which is based on the complex event processing technology framework. The system can handle a large amount of sensor data, perform filtering, carry out complex data processing and decision making. The experimental results show that the proposed street lamp fault diagnosis system based on extreme learning machine can diagnose the street lamp fault effectively and respond to it automatically.

Hierarchical data fusion for Smart Healthcare

The Internet of Things (IoT) facilitates creation of smart spaces by converting existing environments into sensor-rich data-centric cyber-physical systems with an increasing degree of automation, giving rise to Industry 4.0. When adopted in commercial/industrial contexts, this trend is revolutionising many aspects of our everyday life, including the way people access and receive healthcare services. As we move towards Healthcare Industry 4.0, the underlying IoT systems of Smart Healthcare spaces are growing in size and complexity, making it important to ensure that extreme amounts of collected data are properly processed to provide valuable insights and decisions according to requirements in place. This paper focuses on the Smart Healthcare domain and addresses the issue of data fusion in the context of IoT networks, consisting of edge devices, network and communications units, and Cloud platforms. We propose a distributed hierarchical data fusion architecture, in which different data sources are combined at each level of the IoT taxonomy to produce timely and accurate results. This way, mission-critical decisions, as demonstrated by the presented Smart Healthcare scenario, are taken with minimum time delay, as soon as necessary information is generated and collected. The proposed approach was implemented using the Complex Event Processing technology, which natively supports the hierarchical processing model and specifically focuses on handling streaming data ‘on the fly’—a key requirement for storage-limited IoT devices and time-critical application domains. Initial experiments demonstrate that the proposed approach enables fine-grained decision taking at different data fusion levels and, as a result, improves the overall performance and reaction time of public healthcare services, thus promoting the adoption of the IoT technologies in Healthcare Industry 4.0.