Event Processing Systems Research Articles

Thingsonomy: Tackling Variety in Internet of Things Events

The Internet of Things (IoT) will connect billions of devices to the Internet and create a large-scale dynamic and open environment with high heterogeneity. To assure rapid adoption of IoT applications, application developers and users need to be abstracted from IoT infrastructure via scalable middleware. Event-processing systems have the potential to contribute in filling the gap between the IoT infrastructure and applications layers. Event processing follows a decoupled model of interaction in space, time, and synchronization. However, the dimension of semantic coupling still exists and poses a challenge to scalability in highly semantically heterogeneous and dynamic environments such as the IoT. Here, the authors describe an approach based on loosely coupled producers and consumers enabled with approximate semantic matching of events. They emphasize a practitioner perspective to IoT architectures for building software that can tackle heterogeneity of IoT events.

An efficient complex event processing system having the ability of parallel processing and multi event pattern sharing

Complex event processing (CEP) is a promising technology, which can evaluate the business logic from a large number of continuous real-time data streams. There are a few issues that are still remai...

Value-driven uncertainty-aware data processing for an RFID-enabled mixed-model assembly line

The use of radiofrequency identification (RFID) technology generates a high-volume, simple and unreliable data stream due to the technology’s inherent unreliability. Such a data stream cannot be directly used for applications, as doing so would lead to inaccurate and unreliable result. In this paper, we propose a value-driven uncertainty-aware data-processing method that considers RFID detection reliability, timeliness and the throughput of an assembly line to characterize the potential benefits of RFID implementation in a mixed-model assembly system. The proposed method includes three components: a complex event processing system, a Bayesian inference model and a value-driven optimization model. We then demonstrate the use of the method by analyzing an automotive mixed-model assembly line. Some insights into management techniques are offered based on a comparison with several existing barcode implementations. The results of the method application also demonstrate that data uncertainty cannot be ignored in RFID cost–benefit analysis. Besides decreasing the cost of RFID technology, improving reading reliability and designing a sophisticated network can also offer significant benefits.

An efficient complex event processing system having the ability of parallel processing and multi event pattern sharing

Complex event processing (CEP) is a promising technology, which can evaluate the business logic from a large number of continuous real-time data streams. There are a few issues that are still remaining, such as linearization of processing realization mode and event pattern description, which would severely affect the detection efficiency of CEP. This paper proposes a CEP system that can efficiently share the associated event pattern and simultaneously process multiple event sources. First, an event presentation model and a complete set of event operators are summarized, on which a multi event pattern specification language with parallel output capability is proposed. Then, an extended colored Petri-net (CEP-PNm) is defined, which can effectively realize the proposed language and share the same nodes between event patterns in a variety of connection types. At last, by comparison with the traditional CEP system, a parallel processing system called PM-CEPs is introduced. The experimental results show that PM-CEPs can improve complex event processing performance horizontally and vertically through event pattern share and parallel processing.

LogCEP - Complex Event Processing based on Pushdown Automaton

Complex (or Composite) event processing systems have become more popular in a number of areas. Non-deterministic finite automata (NFA) are frequently used to evaluate CEP queries. However, it is complex or difficult to use the traditional NFA-based method to process patterns with conjunction and negation. In this paper, we proposed a new CEP system LogCEP using pushdown automaton to support efficient processing of conjunction and negation. First, the semantic and query language specification of LogCEP system are presented. Then, an automaton named LogPDA is proposed for query processing in LogCEP system. LogPDA construction method describes how to convert a query to LogPDA automation. The LogPDA execution approach describes how to detect the specified pattern using LogPDA. Meanwhile, most of previous NFA-based optimizations can be employed to improve the evaluation efficiency. Finally, our simulation based experimental results show that our method not only extended the expressibility and processing capability but also didn't lead to efficiency decreasing.

Optimizing Expressibility and Performance of Kleene Operators on Binary Tree-based Directed Graph in Complex Event Processing

This study presents a novel method to develop kleene operators on a Binary Tree-Based Directed Graph (BTDG) to improve expressibility and performance of our developed complex event processing system. Complex Event Processing (CEP) systems are widely employed to notify opportunities and threats, which many of these situations are discovered through detection of multiple occurrences of the similar set of events. Many of CEP systems proposed kleene operators to detect these situations. The high expressibility of our implemented CEP system achieved by developing algorithms on kleene operator. These algorithms are designed on a BTDG as a unified basis to arrange input data and joining this input according to a given query. The performance is improved through directing events instantly to join with other events, dynamically switching to an optimal plan and late applying combination for kleene with specified size. The experiments shows kleene operator on BTDG brings rather high degree of expressiblity and performance in compare to its counterparts.

Approximate Semantic Matching of Events for the Internet of Things

Event processing follows a decoupled model of interaction in space, time, and synchronization. However, another dimension of semantic coupling also exists and poses a challenge to the scalability of event processing systems in highly semantically heterogeneous and dynamic environments such as the Internet of Things (IoT). Current state-of-the-art approaches of content-based and concept-based event systems require a significant agreement between event producers and consumers on event schema or an external conceptual model of event semantics. Thus, they do not address the semantic coupling issue. This article proposes an approach where participants only agree on a distributional statistical model of semantics represented in a corpus of text to derive semantic similarity and relatedness. It also proposes an approximate model for relaxing the semantic coupling dimension via an approximation-enabled rule language and an approximate event matcher. The model is formalized as an ensemble of semantic and top- k matchers along with a probability model for uncertainty management. The model has been empirically validated on large sets of events and subscriptions synthesized from real-world smart city and energy management systems. Experiments show that the proposed model achieves more than 95% F 1 Score of effectiveness and thousands of events/sec of throughput for medium degrees of approximation while not requiring users to have complete prior knowledge of event semantics. In semantically loosely-coupled environments, one approximate subscription can compensate for hundreds of exact subscriptions to cover all possibilities in environments which require complete prior knowledge of event semantics. Results indicate that approximate semantic event processing could play a promising role in the IoT middleware layer.

A knowledge-based platform for Big Data analytics based on publish/subscribe services and stream processing

Big Data analytics is considered an imperative aspect to be further improved in order to increase the operating margin of both public and private enterprises, and represents the next frontier for their innovation, competition, and productivity. Big Data are typically produced in different sectors of the above organizations, often geographically distributed throughout the world, and are characterized by a large size and variety. Therefore, there is a strong need for platforms handling larger and larger amounts of data in contexts characterized by complex event processing systems and multiple heterogeneous sources, dealing with the various issues related to efficiently disseminating, collecting and analyzing them in a fully distributed way.In such a scenario, this work proposes a way to overcome two fundamental issues: data heterogeneity and advanced processing capabilities. We present a knowledge-based solution for Big Data analytics, which consists in applying automatic schema mapping to face with data heterogeneity, as well as ontology extraction and semantic inference to support innovative processing. Such a solution, based on the publish/subscribe paradigm, has been evaluated within the context of a simple experimental proof-of-concept in order to determine its performance and effectiveness.

A stepwise approach to developing staged applications

The staged event-driven architecture (SEDA) can be seen as a milestone as regards integration of threads and events in a single model. By decomposing applications into sets of multi-threaded stages connected by event queues, SEDA allows for the use of each concurrency model where most appropriate. Inside each SEDA stage, the number and scheduling policy of threads can be adjusted to enhance performance. SEDA lends itself to parallelization on multi-cores and is well suited for many high-volume data stream processing systems and highly concurrent event processing systems. In this paper, we propose an extension to the staged model that decouples application design from specific execution environments, encouraging a stepwise approach for designing concurrent applications, similar to Foster's PCAM methodology. We also present Leda, a platform that implements this extended model. In Leda, stages are defined purely by their role in application logic, with no concern for locality of execution, and are bound together through asynchronous communication channels, called connectors, to form a directed graph representing the flow of events inside the application. Decisions about the configuration of the application at execution time are delayed to later phases of the implementation process. Stages in the application graph can then be grouped to form clusters, and each cluster is mapped to an exclusive OS process, running on an arbitrary host. Finally, we discuss two example applications which we developed to evaluate the Leda platform.

An analysis of performance factors on Esper‐based stream big data processing in a virtualized environment

SUMMARYTo process big data in a cloud computing environment, a large scale of stream big data is one of the challenging issues. In particular, complex event processing is an emerging technology to handle massive events from multiple sources and process them in a near real time. In this paper, we analyze four performance‐influencing factors on a virtualized event processing system: the number of query statements, garbage collection intervals, the quantity of virtual machine resources, and virtual CPU assignment types. In our experiments, we observe the performance effects of these performance parameters, by implementing and running an Esper‐based event processing application on top of a Xen‐based virtualized system. With experimental results, we analyze the memory consumption problem, and apply periodic garbage collection, to reduce unexpected memory consumption of JVM. Also, we analyze performance effects of the number of cores in a virtual machine (VM) and resource sharing among VMs. Under the virtualized environment in cloud computing infrastructure, one of the critical issues is the management of virtualized computing resources. Accordingly, we present the event processing performance on VMs as a function of virtual CPU assignment types and the number of VMs that share virtual CPUs. Copyright © 2014 John Wiley & Sons, Ltd.

JEPC: The Java Event Processing Connectivity

Today, event processing (EP) is the first choice technology for analyzing massive event streams in a timely manner. EP allows to detect user-defined situations of interest, like in streaming position events for example, in near real-time such that actions can be taken immediately. Unfortunately, each specific EP system has its very own API and query language because there are no standards. The exchange of EP systems as well as their use within a federation is challenging, error-prone, and expensive. To overcome these problems, we introduce the Java Event Processing Connectivity (JEPC) that is a middleware for uniform EP functionality in Java. JEPC provides always the same API and query language for EP completely independent of the EP system beneath. Furthermore, we show in detail how JEPC can integrate database systems besides EP systems and evaluate the performance of EP powered by databases systems.

Towards pattern‐based architectures for event processing systems

SUMMARYRecently, event processing (EP) has gained considerable attention as an individual discipline in computer science. From a software engineering perspective, EP systems still lack the maturity of well‐established software architectures. For the development of industrial EP systems, generally accepted software architectures based on proven design patterns and principles are still missing. In this article, we introduce a catalog of design patterns that supports the development of event‐driven architectures (EDAs) and complex EP systems. The design principles originate from experiences reported in publications as well as from our own experiences in building EP systems with industrial and academic partners. We present several patterns on different layers of abstractions that define the overall structure as well as the building blocks for EP systems. Architectural patterns that determine the top‐level structure of an EDA can be distinguished from design patterns that specify the basic mechanisms of EP. The practical application of the catalog of patterns is described by the pattern‐based design of a sample EDA for a sensor‐based energy control system. Finally, we propose a coherent and general reference architecture for EP derived from the proposed patterns.Copyright © 2013 John Wiley & Sons, Ltd.

비정형 빅데이터의 실시간 복합 이벤트 탐지를 위한 기법

최근 소셜 미디어의 발달과 스마트폰의 확산으로 SNS(Social Network Service)가 활성화가 되면서 데이터양이 폭발적으로 증가하였다. 이에 맞춰 빅데이터 개념이 새롭게 대두되었으며, 빅데이터를 활용하기 위한 많은 방안이 연구되고 있다. 여러 기업이 보유한 빅데이터의 가치창출을 극대화하기 위해 기존 데이터와의 융합이 필요하며, 물리적, 논리적 저장구조가 다른 이기종 데이터 소스를 통합하고 관리하기 위한 시스템이 필요하다. 빅데이터를 처리하기 위한 시스템인 맵리듀스는 분산처리를 활용하여 빠른게 데이터를 처리한다는 이점이 있으나 모든 키워드에 대해 시스템을 구축하여 저장 및 검색 등의 과정을 거치므로 실시간 처리에 어려움이 따른다. 또한, 이기종 데이터를 처리하는 구조가 없어 복합 이벤트를 처리하는데 추가 비용이 발생할 수 있다. 이를 해결하는 방안으로 기존에 연구된 복합 이벤트 처리 시스템을 활용하여 실시간 복합 이벤트 탐지를 위한 기법을 제안하고자 한다. 복합 이벤트 처리 시스템은 서로 다른 이기종 데이터 소스로부터 각각의 데이터들을 통합하고 이벤트들의 조합이 가능하며 스트림 데이터를 즉시 처리할 수 있어 실시간 처리에 유용하다. 그러나 SNS, 인터넷 기사 등 텍스트 기반의 비정형 데이터를 텍스트형으로 관리하고 있어 빅데이터에 대한 질의가 요청될 때마다 문자열 비교를 해야 하므로 성능저하가 발생할 여지가 있다. 따라서 복합 이벤트 처리 시스템에서 비정형 데이터를 관리하고 질의처리가 가능하도록 문자열의 논리적 스키마를 부여하고 데이터 통합 기능을 제안한다. 그리고 키워드 셋을 이용한 필터링 기능으로 문자열의 키워드를 정수형으로 변환함으로써 반복적인 비교 연산을 줄인다. 또한, 복합 이벤트 처리 시스템을 활용하면 인 메모리(In-memory)에서 실시간 스트림 데이터를 처리함으로써 디스크에 저장하고 불러들이는 시간을 줄여 성능 향상을 가져온다. Recently, due to the growth of social media and spread of smart-phone, the amount of data has considerably increased by full use of SNS (Social Network Service). According to it, the Big Data concept is come up and many researchers are seeking solutions to make the best use of big data. To maximize the creative value of the big data held by many companies, it is required to combine them with existing data. The physical and theoretical storage structures of data sources are so different that a system which can integrate and manage them is needed. In order to process big data, MapReduce is developed as a system which has advantages over processing data fast by distributed processing. However, it is difficult to construct and store a system for all key words. Due to the process of storage and search, it is to some extent difficult to do real-time processing. And it makes extra expenses to process complex event without structure of processing different data. In order to solve this problem, the existing Complex Event Processing System is supposed to be used. When it comes to complex event processing system, it gets data from different sources and combines them with each other to make it possible to do complex event processing that is useful for real-time processing specially in stream data. Nevertheless, unstructured data based on text of SNS and internet articles is managed as text type and there is a need to compare strings every time the query processing should be done. And it results in poor performance. Therefore, we try to make it possible to manage unstructured data and do query process fast in complex event processing system. And we extend the data complex function for giving theoretical schema of string. It is completed by changing the string key word into integer type with filtering which uses keyword set. In addition, by using the Complex Event Processing System and processing stream data at real-time of in-memory, we try to reduce the time of reading the query processing after it is stored in the disk.

User-centric modeling and processing for ubiquitous events using semantic capability models

Recent development in sensor networks and mobile computing is gaining increasing interest from enterprises. Sensor data services can provide fine-grained information of the physical world and feed the event processing systems where high level business logic is evaluated and coarse-grained business events are derived. However current event processing systems are not user-oriented and take considerable effort to configure and implement mainly because of the granularity mismatch. In this paper, we present a intuitive way to model the complex event patterns based on semantic descriptions of sensor service capabilities. Then, we transform the event patterns into stream queries to facilitate an automatic implementation for the event processing system.

A simulation process for asynchronous event processing systems: Evaluating performance and availability in transaction models

Simulation is essential for understanding the performance and availability behavior of complex systems, but there are significant difficulties when trying to simulate systems with multiple components, which interact with asynchronous communication. A systematic process is needed, in order to cope with the complexity of asynchronous event processing and the failure semantics of the interacting components. We address this problem by introducing an approach that combines formal techniques for faithful representation of the complex system effects and a statistical analysis for simultaneously studying multiple simulation outcomes, in order to interpret them. Our process has been successfully applied to a synthetic workload for distributed transaction processing. We outline the steps followed towards generating a credible simulation model and subsequently we report and interpret the results of the applied statistical analysis. This serves as a proof of concept that the proposed simulation process can be also effective in other asynchronous system contexts, like for example distributed group communication systems, file systems and so on.

A lightweight and extensible Complex Event Processing system for sense and respond applications

Complex Event Processing (CEP) is considered as a promising technology for enabling the evolution of service-oriented enterprise systems towards operational aware systems. CEP effectively supports the implementation of “sense and respond” behaviours, as it enables to extract meaningful events from raw data streams originated by sensing infrastructures, for enterprise processes and applications consumption. This paper proposes a novel CEP engine conceived with ease of use, extensibility, portability, and scalability requirements in mind. More specifically, we propose a Lightweight Stage-based Event Processor (LiSEP) based on a layered architectural design. Thanks to the adoption of Staged Event-Driven Architecture principles, core event processing logic is decoupled from low-level thread management issues. This results in an easy-to-understand and extensible implementation while testing results show performance scalability. We report on the carrying out of a case study on dangerous goods monitoring in maritime transport. The objective of the case study is to develop a Proof of Concept application leveraging on LiSEP capabilities in sensor and RFID events processing for monitoring and alerting purposes.

An Optimization Method for Sequence Event Detection

Composite event processing systems are useful in various application domains such as stock data stream monitoring, RFID data management, web access pattern monitoring, etc. The tree-based method is a typical method for implementing composite event processing system. In this paper, we introduce an optimization method called “condensed composition” for the tree-based composite event processing. “Condensed composition” divides the events in groups to participate the composition. We analyze the reason why the performance is improved and propose the idea of using the statistics of the “condensation degree” to dynamically decide using condensed composition or not for different data characteristics. In the experiments, the “condensed composition” method significantly improved the performance on the real stock data.

Complex event detection at wire speed with FPGAs

Complex event detection is an advanced form of data stream processing where the stream(s) are scrutinized to identify given event patterns. The challenge for many complex event processing (CEP) systems is to be able to evaluate event patterns on high-volume data streams while adhering to real-time constraints. To solve this problem, in this paper we present a hardware-based complex event detection system implemented on field-programmable gate arrays (FPGAs). By inserting the FPGA directly into the data path between the network interface and the CPU, our solution can detect complex events at gigabit wire speed with constant and fully predictable latency, independently of network load, packet size, or data distribution. This is a significant improvement over CPU-based systems and an architectural approach that opens up interesting opportunities for hybrid stream engines that combine the flexibility of the CPU with the parallelism and processing power of FPGAs.

Preventing Future Oil Spills with Software-Based Event Detection

Complex event processing systems detect problems in mission-critical, real-time applications and generate intelligent decisions to modulate the system environment.

Complex Sensor Event Processing for Business Process Integration

Ubiquitous technologies such as sensor network and RFID have enabled companies to realize more rapid and agile manufacturing and service systems. In this paper, we addresses how the huge amount of real-time events coming from these devices can be filtered and integrated to business process such as manufacturing, logistics, and supply chain process. In particular, we focus on complex event processing of sensor and RFID events in order to integrate them to business rules in business activities. We also illustrate a ubiquitous event processing system, named ueFilter, which helps to filter and aggregate sensor event, to detect event patterns from sensors and RFID by means of event pattern languages (EPL), and trigger event-condition-action (ECA) in logistics processes.