Event Stream Processing Systems Research Articles

Formal verification for event stream processing: Model checking of BeepBeep stream processing pipelines

Event stream processing (ESP) is the application of a computation to a set of input sequences of arbitrary data objects, called “events”, in order to produce other sequences of data objects. In recent years, a large number of ESP systems have been developed; however, none of them is easily amenable to a formal verification of properties on their execution. In this paper, we show how stream processing pipelines built with an existing ESP library called BeepBeep 3 can be exported as a Kripke structure for the NuXmv model checker. This makes it possible to formally verify properties on these pipelines, and opens the way to the use of such pipelines directly within a model checker as an extension of its specification language.

Dynamic Performance Analysis of STEP System in Internet of Vehicles Based on Queuing Theory.

The Internet of vehicles (IoV) is an important research area of the intelligent transportation systems using Internet of things theory. The complex event processing technology is a basic issue for processing the data stream in IoV. In recent years, many researchers process the temporal and spatial data flow by complex event processing technology. Spatial Temporal Event Processing (STEP) is a complex event query language focusing on the temporal and spatial data flow in Internet of vehicles. There are four processing models of the event stream processing system based on the complex event query language: finite automata model, matching tree model, directed acyclic graph model, and Petri net model. In addition, the worst-case response time of the event stream processing system is an important indicator of evaluating the performance of the system. Firstly, this paper proposed a core algorithm of the temporal and spatial event stream processing program based on STEP by Petri net model. Secondly, we proposed a novel method to estimate the worst-case response time of the event stream processing system, which is based on stochastic Petri net and queuing theory. Finally, through the simulation experiment based on queuing theory, this paper proves that the data stream processing system based on STEP has good dynamic performance in processing the spatiotemporal data stream in Internet of vehicles.

Serverless Stream Processing with Elastic Multi-M/M/s/K Queue System

The high throughput - low latency stream processing systems are required to be elastic enough to scale for varying load spike on-demand. However, in the current stream processing systems, the load shedding is observed which impacts the final accuracy. In order to get rid of this issue, the elasticity can be implemented in all kinds of resources involved in the stream processing systems. This paper focuses on providing the elastic scalability in queues and Serverless functions for the event stream processing systems. First, we explain the need of elastic multi-queue with Serverless function in detail for event stream processing, and then will propose an algorithm for elastic scalability of multi-M/M/s/K Queuing with Serverless functions for the efficient stream processing. The experiment result shows that the system scales very well in short span of time with the help of our proposed algorithm. The increased availability in turn helps improving the high processing throughput in low latency.

A QoS-Latency Aware Event Stream Processing with Elastic-FaaS

Stream processing systems need to be elastically scalable to process and respond the unpredictable massive load spike in real-time with high throughput and low latency. Though the modern cloud technologies can help in elastically provisioning the required computing resources on-the-fly, finding out the right point-in-time varies among systems based on their expected QoS characteristics. The latency sensitivity of the stream processing applications varies based on their nature and pre-set requirements. For few applications, even a little latency in the response will have huge impact, whereas for others the little latency will not have that much impact. For the former ones, the processing systems are expected to be highly available, elastically scalable, and fast enough to perform, whenever there is a spike. The time required to elasticity provision the systems under FaaS is very high, comparing to provisioning the Virtual Machines and Containers. However, the current FaaS systems have some limitations that need to be overcome to handle the unexpected spike in real-time. This paper proposes a new algorithm called Elastic-FaaS on top of the existing FaaS to overcome this QoS latency issue. Our proposed algorithm will provision required number of FaaS container instances than any typical FaaS can provision normally, whenever there is a demand to avoid the latency issue. We have experimented our algorithm with an event stream processing system and the result shows that our proposed Elastic-FaaS algorithm performs better than typical FaaS by improving the throughput that meets the high accuracy and low latency requirements.

An Adaptive Parallel Processing Strategy for Complex Event Processing Systems over Data Streams in Wireless Sensor Networks.

Efficient matching of incoming events of data streams to persistent queries is fundamental to event stream processing systems in wireless sensor networks. These applications require dealing with high volume and continuous data streams with fast processing time on distributed complex event processing (CEP) systems. Therefore, a well-managed parallel processing technique is needed for improving the performance of the system. However, the specific properties of pattern operators in the CEP systems increase the difficulties of the parallel processing problem. To address these issues, a parallelization model and an adaptive parallel processing strategy are proposed for the complex event processing by introducing a histogram and utilizing the probability and queue theory. The proposed strategy can estimate the optimal event splitting policy, which can suit the most recent workload conditions such that the selected policy has the least expected waiting time for further processing of the arriving events. The proposed strategy can keep the CEP system running fast under the variation of the time window sizes of operators and the input rates of streams. Finally, the utility of our work is demonstrated through the experiments on the StreamBase system.

Aging-related performance anomalies in the apache storm stream processing system

Event stream processing has recently emerged as a popular paradigm for implementing high-volume distributed (near-)real time data processing applications. Several open source systems are today available, supporting the development of such applications, many of which developed with the technologies of the Apache Software Foundation. These so called stream processors are long-running complex software systems which may be affected by software aging, a well-known phenomenon among operation engineers, consisting of a progressive increase in the failure rate or in performance degradation of a software system over time.We address the problem of identifying symptoms and sources of software aging in the Apache Storm event stream processing system; this helps to identify proper strategies to prevent or mitigate anomalous behaviors in production environments. To this aim, we present an experimental study investigating aging manifestations in a popular system, namely Apache Storm. Results show that Storm presents anomalous behaviors in long runs, which prevent some topologies from working continuously. These can be attributed to software aging, due to Storm internal resource management mechanisms influenced by the garbage collector and the memory assigned to worker processes. We discuss the aging-related Apache Storm behaviors, and we experiment rejuvenation actions, showing that they are actually able to remove them.

Attributed Graph Rewriting for Complex Event Processing Self-Management

The use of Complex Event Processing (CEP) and Stream Processing (SP) systems to process high-volume, high-velocity Big Data has renewed interest in procedures for managing these systems. In particular, self-management and adaptation of runtime platforms have been common research themes, as most of these systems run under dynamic conditions. Nevertheless, the research landscape in this area is still young and fragmented. Most research is performed in the context of specific systems, and it is difficult to generalize the results obtained to other contexts. To enable generic and reusable CEP/SP system management procedures and self-management policies, this research introduces the Attributed Graph Rewriting for Complex Event Processing Management ( AGeCEP ) formalism. AGeCEP represents queries in a language- and technology-agnostic fashion using attributed graphs. Query reconfiguration capabilities are expressed through standardized attributes, which are defined based on a novel classification of CEP query operators. By leveraging this representation, AGeCEP also proposes graph rewriting rules to define consistent reconfigurations of queries. To demonstrate AGeCEP feasibility, this research has used it to design an autonomic manager and to define a selected set of self-management policies. Finally, experiments demonstrate that AGeCEP can indeed be used to develop algorithms that can be integrated into diverse CEP systems.

복합 이벤트 스트림 질의 처리 성능 개선을 위한 질의 전처리 도구

A complex event processing system, becoming useful in real life domains, efficiently processes stream of continuous events like sensor data from IoT systems. However, those systems do not work well on some types of queries yet, so that programmers should be careful about that. For instance, they do not sufficiently provide detailed guide to choose efficient queries among the almost same meaning queries. In this paper, we propose an query preprocessing tool for event stream processing systems, which helps programmers by giving them the hints to improve performance whenever their queries fall in any possible bad formats in the performance sense. We expect that our proposed module would be a big help to increases productivity of writing programs where debugging, testing, and performance tuning are not straightforward.

Data Stream Management System and Capital Market Surveillance

Importance of real-time data analysis has been felt since early '90s and thus processing of streaming data (from either sensor networks or telecom switches or web and other disparate systems) is the demand of the industries worldwide. Quicker detection of fraudulent activities in a financial system is the order of the day. Thus capital market surveillance, if can be performed by using the streaming input of various trading transactions, without being stored, that would be beneficial to the regulatory authorities and stock exchanges. In this paper, we describe how stream processing using a data stream management system (DSMS) can be used for the above task and how effective would be that in terms of performance and latency. We present results obtained from using a commercial event stream processing system (IBM InfoSphere Streams platform) for certain typical fraud detection scenarios.

Development of an Event Stream Processing System for the Vehicle Telematics Environment

In this letter, we present an event stream processing system that can evaluate a pattern query for a data sequence with predicates. We propose a pattern query language and develop a pattern query processing system. In our system, we propose novel techniques for run-time aggregation and negation processing and apply our system to stream data generated from vehicles to monitor unusual driving patterns.

Microsoft CEP server and online behavioral targeting

In this demo, we present the Microsoft Complex Event Processing (CEP) Server, Microsoft CEP for short. Microsoft CEP is an event stream processing system featured by its declarative query language and its multiple consistency levels of stream query processing. Query composability, query fusing, and operator sharing are key features in the Microsoft CEP query processor. Moreover, the debugging and supportability tools of Microsoft CEP provide visibility of system internals to users. Web click analysis has been crucial to behavior-based online marketing. Streams of web click events provide a typical workload for a CEP server. Meanwhile, a CEP server with its processing capabilities plays a key role in web click analysis. This demo highlights the features of Microsoft CEP under a workload of web click events.