Microsoft StreamInsight Research Articles

Performance Evaluation of Microsoft StreamInsight as a Complex Events Processing Platform

Internet of Things (IoT) is one of the most-hyped emerging areas of technology in recent times. With things communicating with each other over the internet, it is only natural to expect that the volumes of the data collected and transmitted will become massive. Therefore, analysing such explosive amounts of data in near real-time, to generate actionable information out of these, is an imperative. Complex Event Processing (CEP) refers to the technology that enables near real-time processing of such data streams, without having the need to store and retrieve them from databases. Microsoft StreamInsight is a product that enables Complex Event Processing. In this paper, we talk about how Microsoft StreamInsight can be used to quickly develop a flexible high performance CEP solution. We have given an overview of “IStreamAnalytics”, a platform that we have built using StreamInsight. Further, the performance of StreamInsight as a CEP solution is evaluated through a series of experiments. Keywords: Microsoft StreamInsight, Complex Events Processing, Internet of Things

Online visualization of geospatial stream data using the worldwide telescope

This demo presents the ongoing effort to meld the stream query processing capabilities of Microsoft StreamInsight with the visualization capabilities of the WorldWide Telescope. This effort provides visualization opportunities to manage, analyze, and process real-time information that is of spatio-temporal nature. The demo scenario is based on detecting, tracking and predicting interesting patterns over historical logs and real-time feeds of earthquake data.

Geospatial stream query processing using Microsoft SQL Server StreamInsight

Microsoft SQL Server spatial libraries contain several components that handle geometrical and geographical data types. With advances in geo-sensing technologies, there has been an increasing demand for geospatial streaming applications. Microsoft SQL Server StreamInsight ( StreamInsight , for brevity) is a platform for developing and deploying streaming applications that run continuous queries over high-rate streaming events. With its extensibility infrastructure, StreamInsight enables developers to integrate their domain expertise within the query pipeline in the form of user defined modules. This demo utilizes the extensibility infrastructure in Microsoft StreamInsight to leverage its continuous query processing capabilities in two directions. The first direction integrates SQL spatial libraries into the continuous query pipeline of StreamInsight. StreamInsight provides a well-defined temporal model over incoming events while SQL spatial libraries cover the spatial properties of events to deliver a solution for spatiotemporal stream query processing. The second direction extends the system with an analytical refinement and prediction layer. This layer analyzes historical data that has been accumulated and summarized over the years to refine, smooth and adjust the current query output as well as predict the output in the near future. The demo scenario is based on transportation data in Los Angeles County.

High-performance dynamic pattern matching over disordered streams

Current pattern-detection proposals for streaming data recognize the need to move beyond a simple regular-expression model over strictly ordered input. We continue in this direction, relaxing restrictions present in some models, removing the requirement for ordered input, and permitting stream revisions (modification of prior events). Further, recognizing that patterns of interest in modern applications may change frequently over the lifetime of a query, we support updating of a pattern specification without blocking input or restarting the operator. Our new pattern operator (called AFA) is a streaming adaptation of a non-deterministic finite automaton (NFA) where additional schema-based user-defined information, called a register , is accessible to NFA transitions during execution. AFAs support dynamic patterns, where the pattern itself can change over time. We propose clean order-agnostic pattern-detection semantics for AFAs, with new algorithms that allow a very efficient implementation, while retaining significant expressiveness and supporting native handling of out-of-order input, stream revisions, dynamic patterns, and several optimizations. Experiments on Microsoft StreamInsight show that we achieve event rates of more than 200K events/sec (up to 5x better than simpler schemes). Our dynamic patterns give up to orders-of-magnitude better throughput than solutions such as operator restart, and our other optimizations are very effective, incurring low memory and latency.