Complex Event Processing Engine Research Articles

Building complex event processing capability for intelligent environmental monitoring

Rapid evolution of Internet-of-Things is driving the increased deployment of smart sensors in environmental applications, contributing to many big data characteristics of environmental monitoring. Most of the current environmental monitoring systems are not designed to handle real-time datastreams, and the best practices for datastream processing and predictive analytics are yet to be established. This work presents a complex event processing (CEP) engine for detecting anomalies in real time, and demonstrates it using a series of real monitoring data from the geological carbon sequestration domain. We show that the service-based CEP engine is instrumental for enabling environmental intelligent monitoring systems to ingest heterogeneous datastreams with scalable performance. Our CEP framework requires minimal coding from the user and can be easily extended to other similar environmental monitoring applications.

A Universal Complex Event Processing Mechanism Based on Edge Computing for Internet of Things Real-Time Monitoring

The tremendous number of sensors and smart objects deployed in the Internet of Things (IoT) pose a huge potential for the IoT real-time monitoring applications to detect and react to the real world. The insufficient capacity of the IoT data real-time processing has hampered the growth of the IoT real-time monitoring applications. We focus on two issues of the IoT data real-time processing: 1) how to efficiently transform a large number of raw sensing data into meaningful complex event, and 2) how to adapt to the complexity and changeability of monitoring business logic. This paper proposes a universal complex event processing (CEP) mechanism for the IoT real-time monitoring. We propose a formalized hierarchical complex event model including raw event, simple event, and complex event, which reduces the complexity of event modeling. The model supports complex time and space semantics to define flexible complex events by a programming way. Based on this model, we propose a CEP system architecture, in which the system is deployed on the network edge between sensing devices in terminal and applications in the cloud. The complex event definition can be mapped to the CEP rule logic script to detect the potential abnormal event timely. The proposed CEP mechanism is universal and suitable to any heterogeneous sensing devices and CEP engine. We demonstrate the efficacy of the mechanism with two application case studies that highlight our proposed complex event model and evaluate the performance improvement with experiments.

Even Driven Multimodal Augmented Reality based Command and Control Systems for Mining Industry

The mining industry is confronted with obligations to improve worker safety and provide more effective maintenance of the new-age mining machineries. It is also essential to develop reconnaissance mission to support emergency rescue operations. An approach to ensure the mine workers safety using AR (Augmented Reality) with remote monitoring CCS (Command Control System) is proposed in this paper. This method provides an interactive and augmented real time instantaneous personal safety solution. The benefit of augmented reality is blended with real time mining environmental sensor data using multimodal learning approach to predict the emergency situation instantaneously. Environmental mining sensory information is processed with Complex Event Processing (CEP) engine to derive high level events that trigger the alarm for the emergency situation. The proposed event driven multimodal AR based CCS, outperforms the existing emergency prediction solutions that use sensor fusion or deep learning AR.

A Language for Online State Processing of Binary Sensors, Applied to Ambient Assisted Living

There is a large variety of binary sensors in use today, and useful context-aware services can be defined using such binary sensors. However, the currently available approaches for programming context-aware services do not conveniently support binary sensors. Indeed, no existing approach simultaneously supports a notion of state, central to binary sensors, offers a complete set of operators to compose states, allows to define reusable abstractions by means of such compositions, and implements efficient online processing of these operators. This paper proposes a new language for event processing specifically targeted to binary sensors. The central contributions of this language are a native notion of state and semi-causal operators for temporal state composition including: Allen's interval relations generalized for handling multiple intervals, and temporal filters for handling delays. Compared to other approaches such as CEP (complex event processing), our language provides less discontinued information, allows less restricted compositions, and supports reusable abstractions. We implemented an interpreter for our language and applied it to successfully rewrite a full set of real Ambient Assisted Living services. The performance of our prototype interpreter is shown to compete well with a commercial CEP engine when expressing the same services.

Model Driven Development Applied to Complex Event Processing for Near Real-Time Open Data.

Nowadays, data are being produced like never before because the use of the Internet of Things, social networks, and communication in general are increasing exponentially. Many of these data, especially those from public administrations, are freely offered using the open data concept where data are published to improve their reutilisation and transparency. Initially, the data involved information that is not updated continuously such as budgets, tourist information, office information, pharmacy information, etc. This kind of information does not change during large periods of time, such as days, weeks or months. However, when open data are produced near to real-time such as air quality sensors or people counters, suitable methodologies and tools are lacking to identify, consume, and analyse them. This work presents a methodology to tackle the analysis of open data sources using Model-Driven Development (MDD) and Complex Event Processing (CEP), which help users to raise the abstraction level utilised to manage and analyse open data sources. That means that users can manage heterogeneous and complex technology by using domain concepts defined by a model that could be used to generate specific code. Thus, this methodology is supported by a domain-specific language (DSL) called OpenData2CEP, which includes a metamodel, a graphical concrete syntax, and a model-to-text transformation to specific platforms, such as complex event processing engines. Finally, the methodology and the DSL have been applied to two near real-time contexts: the analysis of air quality for citizens’ proposals and the analysis of earthquake data.

Join query optimization techniques for complex event processing applications

Complex event processing (CEP) is a prominent technology used in many modern applications for monitoring and tracking events of interest in massive data streams. CEP engines inspect real-time information flows and attempt to detect combinations of occurrences matching predefined patterns. This is done by combining basic data items, also called "primitive events", according to a pattern detection plan, in a manner similar to the execution of multi-join queries in traditional data management systems. Despite this similarity, little work has been done on utilizing existing join optimization methods to improve the performance of CEP-based systems. In this paper, we provide the first theoretical and experimental study of the relationship between these two research areas. We formally prove that the CEP Plan Generation problem is equivalent to the Join Query Plan Generation problem for a restricted class of patterns and can be reduced to it for a considerably wider range of classes. This result implies the NP-completeness of the CEP Plan Generation problem. We further show how join query optimization techniques developed over the last decades can be adapted and utilized to provide practically efficient solutions for complex event detection. Our experiments demonstrate the superiority of these techniques over existing strategies for CEP optimization in terms of throughput, latency, and memory consumption.

Cloud computing and big data: Technologies and applications

Cloud computing and big data: Technologies and applications

A visual programming framework for distributed Internet of Things centric complex event processing

Complex Event Processing (CEP) is a promising approach for real-time processing of big-data streams originating from Internet of Things (IoT) devices. Even though scalability and flexibility are key issues for IoT applications, current studies are mostly based on centralized solutions and restrictive query languages. Moreover, development, deployment and operation of big-data applications require significant amount of technical expertise. Hence, a framework that provides a higher abstraction level programming model and a streamlined execution environment is essential to facilitate widespread user adoption. In this paper, we propose a data-flow based visual programming model that enables development of big-data applications in IoT domain even by users who have limited or no experience in big-data technologies. A scalable CEP engine is also devised to execute the applications developed using this language. A prototype is implemented by using the state-of-the-art big-data and real-time stream processing technologies to evaluate the applicability and usability of the framework.

Behavior Drift Detection Based on Anomalies Identification in Home Living Quantitative Indicators

Home Automation and Smart Homes diffusion are providing an interesting opportunity to implement elderly monitoring. This is a new valid technological support to allow in-place aging of seniors by means of a detection system to notify potential anomalies. Monitoring has been implemented by means of Complex Event Processing on live streams of home automation data: this allows the analysis of the behavior of the house inhabitant through quantitative indicators. Different kinds of quantitative indicators for monitoring and behavior drift detection have been identified and implemented using the Esper complex event processing engine. The chosen solution permits us not only to exploit the queries when run “online”, but enables also “offline” (re-)execution for testing and a posteriori analysis. Indicators were developed on both real world data and on realistic simulations. Tests were made on a dataset of 180 days: the obtained results prove that it is possible to evidence behavior changes for an evaluation of a person’s condition.

Multivariate Sensor Data Analysis for Oil Refineries and Multi-mode Identification of System Behavior in Real-time

Large-scale oil refineries are equipped with mission-critical heavy machinery (boilers, engines, turbines, and so on) and are continuously monitored by thousands of sensors for process efficiency, environmental safety, and predictive maintenance purposes. However, sensors themselves are also prone to errors and failure. The quality of data received from these sensors should be verified before being used in system modeling. There is a need for reliable methods and systems that can provide data validation and reconciliation in real-time with high accuracy. In this paper, we develop a novel method for real-time data validation, gross error detection and classification over multivariate sensor data streams. The validated and high-quality data obtained from these processes is used for pattern analysis and modeling of industrial plants. We obtain sensor data from the power and petrochemical plants of an oil refinery and analyze them using various time-series modeling and data mining techniques that we integrate into a complex event processing engine. Next, we study the computational performance implications of the proposed methods and uncover regimes where they are sustainable over fast streams of sensor data. Finally, we detect shifts among steady-states of data, which represent systems’ multiple operating modes and identify the time when a model reconstruction is required using DBSCAN clustering algorithm.

A Smart IoT-Aware System For Crisis Scenario Management

In most dangerous events, involving many people in large buildings, rescue workers need to intervene in a timely and targeted manner in order to help most number of people and secure the environments without wasting resources. This work presents an Internet of Things(IoT)-based framework, aiming at monitoring environmental parameters in order to alert rescuers when they exceed some alarm thresholds. A hardware infrastructure driven by a software layer adds flexibility and adaptability to the Complex Event Processing engine and to a rule engine-based reflective middleware that manages and analyzes raw data in conjunction with a knowledge base modeling the application domain.

CEP4HFP: Complex Event Processing for Heart Failure Prediction.

This paper presents a novel health analysis approach for heart failure prediction. It is based on the use of complex event processing (CEP) technology, combined with statistical approaches. A CEP engine processes incoming health data by executing threshold-based analysis rules. Instead of having to manually set up thresholds, our novel statistical algorithm automatically computes and updates thresholds according to recorded historical data. Experimental results demonstrate the merits of our approach in terms of speed, precision, and recall.

Performance evaluation of linked stream data processing engines for situational awareness applications

SummaryThe easy deployment of wireless sensors allows the development of context‐aware applications that could react to the environment changes and users preferences. For example, data extracted from mobile phones, IoT devices, and embedded computers in buses and taxis could be processed in real time to understand city dynamics. However, gathering and real‐time processing of relevant data are still a challenging task. For instance, the high volume of streaming data requires real‐time processing in order to generate immediate mitigation actions. Complex‐event processing (CEP) techniques and predictive analytics have been recently proposed for analyzing streaming data in order to generate fast insights and then take suitable actions according to the situations' changes. These techniques could be also used to predict future situations and react to them properly before happening. The work presented in this paper focuses mainly on the performance evaluation of three CEP engines, CQELS, C‐SPARQL, and ETALIS, that are widely used by researchers for linked stream data processing. Experiments have been conducted using two existing benchmarks, CityBench and SP2Bench. Several performance metrics have been evaluated to assess on their ability to process high streaming data using complex queries. Results are reported to show the efficiency and scalability of these CEP engines for both social‐based data (SP2Bench) and physical‐based data (CityBench). Reported results show that ETALIS outperforms CQELS and C‐SPARQL in terms of throughput and memory utilization. ETALIS was integrated as a use case scenario for occupancy and air quality comfort in energy‐efficient buildings. Results show the usefulness of CEP for real‐time data processing and reasoning.

MEdit4CEP-Gam: A model-driven approach for user-friendly gamification design, monitoring and code generation in CEP-based systems

ContextGamification has been proven to increase engagement and motivation in multiple and different non-game contexts such as healthcare, education, workplace, and marketing, among others. However, many of these applications fail to achieve the desired benefits of gamification, mainly because of a poor design. ObjectiveThis paper explores the conceptualization, implementation and monitoring phases of meaningful gamification strategies and proposes a solution for strategy experts that hides the implementation details and helps them focus only on what is crucial for the success of the strategy. The solution makes use of Model-Driven Engineering (MDE) and Complex Event Processing (CEP) technology. MethodAn easy-to-use graphical editor is used to provide the high-level models that represent the design of the gamification strategy and its deployment and monitoring. These models contain the event pattern definitions to be automatically transformed into code. This code is then deployed both in a CEP engine to detect the conditions expressed in such patterns and in an enterprise service bus to execute the corresponding pattern actions. ResultsThe paper reports on the use of both a graphical modeling editor for gamification domain definition and a graphical modeling editor for gamification strategy design, monitoring and code generation in event-based systems. It also shows how the proposal can be used to design and automate the implementation and monitoring of a gamification strategy in an educational domain supported by a well-known Learning Management System (LMS) such as Moodle. ConclusionIt can be concluded that this unprecedented model-driven approach leveraging gamification and CEP technology provides strategy experts with the ability to graphically define gamification strategies, which can be directly transformed into code executable by event-based systems. Therefore, this is a novel solution for bringing CEP closer to any strategy expert, positively influencing the gamification strategy design, implementation and real-time monitoring processes.

COLLECT: COLLaborativE ConText-aware service oriented architecture for intelligent decision-making in the Internet of Things

Abstract Internet of Things (IoT) has radically transformed the world; currently, every device can be connected to the Internet and provide valuable information for decision-making. In spite of the fast evolution of technologies accompanying the grow of IoT, we are still faced with the challenge of providing a service oriented architecture, which facilitates the inclusion of data coming together from several IoT devices, data delivery among a system's agents, real-time data processing and service provision to users. Furthermore, context-aware data processing and architectures still pose a challenge, in spite of being key requirements in order to get stronger IoT architectures. To face this challenge, we propose a COLLaborative ConText Aware Service Oriented Architecture (COLLECT), which facilitates both the integration of IoT heterogeneous domain context data — through the use of a light message broker — and easy data delivery among several agents and collaborative participants in the system — making use of an enterprise service bus —. In addition, this architecture provides real-time data processing thanks to the use of a complex event processing engine as well as services and intelligent decision-making procedures to users according to the needs of the domain in question. As a result, COLLECT has a great impact on context-aware decentralized and collaborative reasoning for IoT, promoting context-aware intelligent decision making in such scope. Since context-awareness is key for a wide range of recommender and intelligent systems, the presented novel solution improves decision making in a large number of fields where such systems require to promptly process a variety of ubiquitous collaborative and context-aware data.

Optimizing SIEM Throughput on the Cloud Using Parallelization.

Processing large amounts of data in real time for identifying security issues pose several performance challenges, especially when hardware infrastructure is limited. Managed Security Service Providers (MSSP), mostly hosting their applications on the Cloud, receive events at a very high rate that varies from a few hundred to a couple of thousand events per second (EPS). It is critical to process this data efficiently, so that attacks could be identified quickly and necessary response could be initiated. This paper evaluates the performance of a security framework OSTROM built on the Esper complex event processing (CEP) engine under a parallel and non-parallel computational framework. We explain three architectures under which Esper can be used to process events. We investigated the effect on throughput, memory and CPU usage in each configuration setting. The results indicate that the performance of the engine is limited by the number of events coming in rather than the queries being processed. The architecture where 1/4th of the total events are submitted to each instance and all the queries are processed by all the units shows best results in terms of throughput, memory and CPU usage.

Knowledge-infused and consistent Complex Event Processing over real-time and persistent streams

Emerging applications in Internet of Things (IoT) and Cyber–Physical Systems (CPS) present novel challenges to Big Data platforms for performing online analytics. Ubiquitous sensors from IoT deployments are able to generate data streams at high velocity, that include information from a variety of domains, and accumulate to large volumes on disk. Complex Event Processing (CEP) is recognized as an important real-time computing paradigm for analyzing continuous data streams. However, existing work on CEP is largely limited to relational query processing, exposing two distinctive gaps for query specification and execution: (1) infusing the relational query model with higher level knowledge semantics, and (2) seamless query evaluation across temporal spaces that span past, present and future events. These allow accessible analytics over data streams having properties from different disciplines, and help span the velocity (real-time) and volume (persistent) dimensions. In this article, we introduce a Knowledge-infused CEP (χ-CEP ) framework that provides domain-aware knowledge query constructs along with temporal operators that allow end-to-end queries to span across real-time and persistent streams. We translate this query model to efficient query execution over online and offline data streams, proposing several optimizations to mitigate the overheads introduced by evaluating semantic predicates and in accessing high-volume historic data streams. In particular, we also address temporal consistency issues that arise during fault recovery of query plans that span the boundary between real-time and persistent streams. The proposed χ-CEP query model and execution approaches are implemented in our prototype semantic CEP engine, SCEPter. We validate our query model using domain-aware CEP queries from a real-world Smart Power Grid application, and experimentally analyze the benefits of our optimizations for executing these queries, using event streams from a campus-microgrid IoT deployment. Our results show that we are able to sustain a processing throughput of 3,000 events/secs for χ-CEP queries, a 30× improvement over the baseline and sufficient to support a Smart Township, and can resume consistent processing within 20 secs after stream outages as long as 2 hours.

A Novel Complex Event Processing Engine for Intelligent Data Analysis in Integrated Information Systems

Novel and effective engines for data analysis in integrated information systems are urgently required by diverse applications, in which massive business data can be analyzed to enable the capturing of various situations in real time. The performance of existing engines has limited capacity of data processing in distributed computing. Although Complex Event Processing (CEP) has enhanced the capacity of data analysis in information systems, it is still a big challenging task since events are rapidly increasing in diverse applications. In this paper, a lightweight intelligent data analysis system with a novel CEP engine named LIDA-E is introduced, which employs the knowledge base with rules and an event processing algorithm for analysis. Event models as well as operators support the rules for event selection and aggregation. These operators and rules have been utilized for constructing new CEP system architecture which combines expressiveness and efficiency in analysis. It adopts the agents and filter conception explicitly to provide the event transmission mechanism efficiently. Finally, the comparison between the proposed engine and the existing engine shows that LIDA-E has 48.65% averagely reduced time cost in different tests. The experimental results demonstrate that the developed architecture has better performance in both transmitting and analyzing a large number of events.

An Intelligent Complex Event Processing withDNumbers under Fuzzy Environment

Efficient matching of incoming mass events to persistent queries is fundamental to complex event processing systems. Event matching based on pattern rule is an important feature of complex event processing engine. However, the intrinsic uncertainty in pattern rules which are predecided by experts increases the difficulties of effective complex event processing. It inevitably involves various types of the intrinsic uncertainty, such as imprecision, fuzziness, and incompleteness, due to the inability of human beings subjective judgment. Nevertheless,Dnumbers is a new mathematic tool to model uncertainty, since it ignores the condition that elements on the frame must be mutually exclusive. To address the above issues, an intelligent complex event processing method withDnumbers under fuzzy environment is proposed based on the Technique for Order Preferences by Similarity to an Ideal Solution (TOPSIS) method. The novel method can fully support decision making in complex event processing systems. Finally, a numerical example is provided to evaluate the efficiency of the proposed method.

Multi-stage Monitoring of Abnormal Situation Based on Complex Event Processing

This paper focuses on the monitoring of abnormal situation in workspace where complicate production activities are performed and possible abnormal situations vary in different stages. The monitoring application should track the production process, identifying the production stage and detecting anomaly in every stage as defined. With the development of ubiquitous computing technology and widespread of sensing equipment, context information pertaining to smart working environment is available for monitoring applications. Complex event processing (CEP) is usually introduced to process and correlate context information for its attractive feature of extracting composite event from a large amount of event data in real time according to user-defined event patterns. In this paper, we present context model and event model in which discrete event such as acquiring context value at a point of time is represented by context. The abnormal situation in every stage of production can be transformed into event expressions, called abnormal event patterns. Contexts in different time captured by sensors form data streams and processed by CEP engine to detect abnormal situation. We propose to use state transition to model each stage so that the normal transition period in the beginning and end of stage can be distinguished from abnormal situation. Once a stage is identified to be starting or ending, the application will change abnormal event patterns accordingly. Case study about metallographic examination proves that the approach we propose is effective and feasible for some multi-stage abnormal situation monitoring.