Data Stream Management Systems Research Articles

The Design of Query Processing in Data Stream Management System

data stream management system is used to manage and query coming large, continuous, fast and flexible data stream. The system is based on the flow of data extraction, transformation, combination, which is the main content and task query execution. This paper mainly discusses the design and implementation of query execution module and query execution is composed of two parts which include query operations, query execution and scheduling.

Geospatial data streams: Formal framework and implementation

A spatio-temporal database manages spatio-temporal objects and supports corresponding query languages. Today, the term moving objects databases is used as a synonym for spatio-temporal databases managing spatial objects with a continuously changing geospatial location and/or extent. Recent advances in wireless communication, miniaturization of spatially enabled devices and global navigation satellite systems (GNSS) services have resulted in a large number of novel application domains. Applications in these novel domains (geo-sensor networks, moving objects tracking, real-time traffic analysis, etc.) process huge volumes of continuous data streams, i.e. data sets that are produced incrementally over time, rather than those available in full before the processing begins. Several data stream management systems (DSMSs) have been developed to manage this data. Since they are mainly based on a relational paradigm, they do not support geospatial data. Therefore, there is an urgent need for geospatial data stream management, ranging from real-time monitoring and alerting to long-term analysis of processed geospatial data. In this paper we present a formal framework consisting of data types and operations needed to support geospatial data in data streams. It can be used as a basis either for implementation of a completely new geospatial DSMS, or for extending available open source products and research prototypes. We leverage the work on abstract data types from spatio-temporal databases, present an implementation based on user-defined aggregate functions and illustrate embedding into an SQL-like language.

Research on Multi-Attribute Sequence Query Processing Techniques over Data Streams

Due to the great progress of computer technology and mature development of network, more and more data are generated and distributed through the network, which is called data streams. During the last couple of years, a number of researchers have paid their attention to data stream management, which is different from the conventional database management. At present, the new type of data management system, called data stream management system (DSMS), has become one of the most popular research areas in data engineering field. Lots of research projects have made great progress in this area. Since the current DSMS does not support queries on sequence data, this paper, we will focus on multi-attribute streams, such as video films. We will discuss the related issues such as how to build an efficient index for all queries of different streams and the corresponding query processing mechanisms.

WD-PWS: THE FIRST SEMANTICS FOR QUERYING OVER PROBABILISTIC DATA STREAMS WITH CONTINUOUS DISTRIBUTIONS

Many emerging applications need continuous querying over uncertain event streams, mostly for online monitoring. These streaming uncertain events may come from radars, sensors, or even software hooks. The uncertainty is usually due to measurement errors, inherent ambiguities and privacy preserving reasons. To cover new requirements, we have designed and implemented a new system called Probabilistic Data Stream Management System (PDSMS) in Ref. 1. PDSMS is a data processing engine which runs continuous queries over probabilistic streams. However, lack of a semantics for probabilistic databases which supports continuous distributions prevented us from having a strong foundation for our query operators. It also precludes us from proving consistency and correctness of query operations especially after optimization and adaption. In fact, in the probabilistic database literature, there is no semantics available which covers continuous distributions. This limitation is very restrictive as in real-world, uncertainty is usually modeled by continuous distributions. In this paper, after presenting a basic probabilistic data model for PDSMS, we focus on querying and formally present the first semantics for probabilistic query operations which supports continuous distributions as well as discrete ones. Using this new semantics, we define our query operators (e.g. select, project, and join) formally without ambiguity and compatible with operators in relational algebra. Thus, we can leverage many transformation rules in relational algebra as well. This new semantics allows us to have different strictness levels and consistency between operators. We also proved many strictness theorems about different alternatives for query operators.

Continuously Moving Objects Framework to Monitor Result Changes of Spatio Temporal Queries using Data Stream Management System

In this paper the concept of safe limit in the framework for continuously moving objects to monitor result changes of spatio temporal queries has been proposed. In this framework the movement of the moving object is monitored with the help of user defined aggregate function over spatio temporal reference in data stream management system. The expected movement of the moving object is plotted on the graph and the spatio temporal queries are answered on the basis of that, until and unless the difference between the expected movement and actual movement is more than the safe limit. This makes the framework more efficient than the previously given framework. The safe limit can be any range with respect to space and time of a moving object which varies according to different parameters such as size of the object, velocity with which it is moving and etc.

Continuously Moving Objects Framework to Monitor Result Changes of Spatio Temporal Queries using Data Stream Management System

In this paper the concept of safe limit in the framework for continuously moving objects to monitor result changes of spatio temporal queries has been proposed. In this framework the movement of the moving object is monitored with the help of user defined aggregate function over spatio temporal reference in data stream management system. The expected movement of the moving object is plotted on the graph and the spatio temporal queries are answered on the basis of that, until and unless the difference between the expected movement and actual movement is more than the safe limit. This makes the framework more efficient than the previously given framework. The safe limit can be any range with respect to space and time of a moving object which varies according to different parameters such as size of the object, velocity with which it is moving and etc.

Similarity based join over audio feeds in a multimedia data stream management system

Over the last several years, processing of high performance data streams has become very important in various domains. A new type of data processing is needed for applications where input data streams are modeled as multimedia data streams, such as audio and video feeds. For example, in the public safety sector, monitoring and automatic identification of particular individuals suspected of terrorist or criminal activity requires the processing of complex audio and video streams, which is beyond the capabilities of a typical data stream management system (DSMS). The concept of a multimedia data stream management system (MMDSMS) has recently been introduced in order to effectively process continuous queries over dynamic multimedia data streams. In this paper, we address MMDSMS functionalities related to speaker recognition problems in the area of detecting individuals who may pose security threats. We focus on audio feed processing using our novel similarity-based join and on parameterization of the multimedia signal for the process of recognition. We propose a set of signal parameters which a clearly discriminate among individual voices by describing the signal using a homomorphic processing method. Our research was primarily focused on assessing the applicability of cepstral analysis in speech recognition systems, based on a set of acquired digitized voice samples. We developed a research prototype to assess the proposed concepts, and verified the effectiveness of our framework in a lab environment.

Generic windowing support for extensible stream processing systems

Stream processing applications process high volume, continuous feeds from live data sources, employ data-in-motion analytics to analyze these feeds, and produce near real-time insights with low latency. One of the fundamental characteristics of such applications is the on-the-fly nature of the computation, which does not require access to disk resident data. Stream processing applications store the most recent history of streams in memory and use it to perform the necessary modeling and analysis tasks. This recent history is often managed using windows. All data stream management systems provide some form of windowing functionality. Windowing makes it possible to implement streaming versions of the traditionally blocking relational operators, such as streaming aggregations, joins, and sorts, as well as any other analytic operator that requires keeping the most recent tuples as state, such as time series analysis operators and signal processing operators.

Research and Application on Data Stream Management System of Expressway Real-Time Monitoring System

This paper mainly puts forward a new idea that using DSMS (data stream management system) in GIS application can improve the performance of GIS application systems. It briefly discusses the necessity of using DSMS, and introduces the related technologies about streaming data. The expressway real-time monitoring system was taken as an example of GIS application. First, it analyzes the functions realized by DSMS in detail, describing basic architecture of the system, focusing on the specific process of streaming data processing. In addition, it discusses the related technology of continuous query language and analyzes specific continuous queries of some functions. Finally, the realization of the functions shows that the expressway real-time monitoring system has a better performance and reaches real-time by using DSMS.

Research on Sequence Query Processing Techniques over Data Streams

Due to the great progress of computer technology and mature development of network, more and more data are generated and distributed through the network, which is called data streams. During the last couple of years, a number of researchers have paid their attention to data stream management, which is different from the conventional database management. At present, the new type of data management system, called data stream management system (DSMS), has become one of the most popular research areas in data engineering field. Lots of research projects have made great progress in this area. Since the current DSMS does not support queries on sequence data, this project will study the issues related to two types of data. First, we will focus on the content filtering on single-attribute streams, such as sensor data. Second, we will focus on multi-attribute streams, such as video films. We will discuss the related issues such as how to build an efficient index for all queries of different streams and the corresponding query processing mechanisms.

An Experimental Evaluation of Distributed Data Stream Processing using Lightweight RDBMS SQLite

The data generated at a very high rate by sensors and RFIDs are required to be handled by continuous queries keeping real time response. DSMSs (Data Stream Management systems) are used in several cases of these large scale systems for its purpose. This paper describes the results of experimental evaluation of distributed data stream processing using SQLite which is a lightweight RDBMS as a stream processing engine node. In particular, it is necessary to correspond also to the increasing data rate flexibly in data stream processing. This paper proved that it could respond to the increase in a data rate by adding a processing node dynamically. In that case, it is necessary to perform synchronous processing in two or more processing nodes. Therefore, within the limits of this experiment, it turned out that it is realizable by performing synchronous processing by the side which divides and passes data.

Multilevel secure data stream processing: Architecture and implementation

The proliferation of sensors and mobile devices and their connectedness to the network have given rise to numerous types of situation monitoring applications. Data Stream Management Systems DSMSs have been proposed to address the data processing needs of such applications that require collection of high-speed data, computing results on-the-fly, and taking actions in real-time. Although a lot of work appears in the area of DSMS, not much has been done in multilevel secure MLS DSMS making the technology unsuitable for highly sensitive applications, such as battlefield monitoring. An MLS--DSMS should ensure the absence of illegal information flow in a DSMS and more importantly provide the performance needed to handle continuous queries. We illustrate why the traditional DSMSs cannot be used for processing multilevel secure continuous queries and discuss various DSMS architectures for processing such queries. We implement one such architecture and demonstrate how it processes continuous queries. In order to provide better quality of service and memory usage in a DSMS, we show how continuous queries submitted by various users can be shared. We provide experimental evaluations to demonstrate the performance benefits achieved through query sharing.

PROBABILISTIC QUERYING OVER UNCERTAIN DATA STREAMS

Inherent imprecision of data in many applications motivates us to support uncertainty as a first-class concept. Data stream and probabilistic data have been recently considered noticeably in isolation. However, there are many applications including sensor data management systems and object monitoring systems which need both issues in tandem. Our main contribution is designing a probabilistic data stream management system, called Sarcheshmeh, for continuous querying over probabilistic data streams. Sarcheshmeh supports uncertainty from input data to final query results. In this paper, after reviewing requirements and applications of probabilistic data streams, we present our new data model for probabilistic data streams and define our main logical operators formally. Then, we present query language and physical operators. In addition, we introduce the architecture of Sarcheshmeh and also describe some major challenges like memory management and our floating precision mechanism toward designing a more robust system. Finally, we report evaluation of our system and the effect of floating precision on the tradeoff between accuracy and efficiency.

Content-based load shedding in multimedia data stream management system

Abstract.Overload management has become very important in public safety systems that analyse high performance multimedia data streams, especially in the case of detection of terrorist and criminal dangers. Efficient overload management improves the accuracy of automatic identification of persons suspected of terrorist or criminal activity without requiring interaction with them. We argue that in order to improve the quality of multimedia data stream processing in the public safety arena, the innovative concept of a Multimedia Data Stream Management System (MMDSMS) using load-shedding techniques should be introduced into the infrastructure to monitor and optimize the execution of multimedia data stream queries. In this paper, we present a novel content-centered load shedding framework, based on searching and matching algorithms, for analysing video tuples arriving within multimedia data streams. The framework tracks and registers all symptoms of overload, and either prevents overload before it occurs, or minimizes its effects. We have extended our Continuous Query Language (CQL) syntax to enable this load shedding technique. The effectiveness of the framework has been verified using both artificial and real data video streams collected from monitoring devices.

Delivering QOS in XML Data Stream Processing Using Load Shedding

In recent years, we have witnessed the emergence of new types of systems that deal with large volumes of streaming data. Examples include financial data analysis on feeds of stock tickers, sensor- based environmental monitoring, network track monitoring and click stream analysis to push customized advertisements or intrusion detection. Traditional database management systems (DBMS), which are very good at managing large volumes of stored data, are not suitable for this, as streaming needs low-latency processing on live data from push-based sources. Data Stream Management Systems (DSMS) are fast emerging to address this new type of data, but faces challenging issues, such as unpredictable data arrival rate. On bursty mode, processing need surpasses available system capacity affecting the Quality of Service (QoS) adversely. The system overloading is even more acute in XML data streams compared to relational streams due to its extra resource requirements for data preparation and result construction. The main focus of this paper is to find out suitable ways to process this high volume of data streams dealing with the spikes in data arrival gracefully, under limited or fixed system resources in the XML stream context.

데이터 스트림에서 다중 조인 연속질의의 효과적인 처리를 위한 전처리 기법

데이터 스트림이란 빠르게 연속적으로 무한히 발생하는 데이터 집합을 의미한다. 최근 다양한 산업의 발달로 인해 이러한 스트림 데이터의 효율적인 처리를 위한 요구 사항들이 늘어나고 있다. 특히 많은 연산 비용을 요구하는 조인 연산의 효율적인 처리는 데이터 스트림 관리 시스템의 성능 향상에 많은 영향을 미친다. 본 논문에서는 다중 조인 연속질의의 효율적인 처리를 위하여 최종 질의 결과에 포함되지 않는 불필요한 중간 조인 결과들을 사전에 제거함으로써 조인 연산의 비용을 감소시키는 방법을 제안한다. 이를 위해 스트림 데이터의 모니터링을 위한 매트릭스 기반의 구조체를 제안하고, 제안된 구조체를 이용한 매트릭스 연산을 통하여 최종 조인 결과의 투플 수를 예측함과 동시에 불필요한 중간 결과들을 만들어내는 투플들을 찾아낸다. 이를 통해 해당 투플을 이용한 조인 연산의 수행 여부를 결정하여 최종 조인 결과를 만들지 않는 투플을 조인 연산에서 배제함으로써 효율적으로 다중 조인 연속 질의를 처리한다. A data stream is a series of tuples which are generated in real-time, incessant, immense, and volatile manner. As new information technologies are actively emerging, stream processing methods are being needed to efficiently handle data streams. Especially, finding out an efficient evaluation for a multi-way join would make outstanding contributions toward improving the performance of a data stream management system because a join operation is one of the most resource-consuming operators for evaluating queries. In this paper, in order to evaluate efficiently a multi-way join continuous query, we propose a novel method to decrease the cost of a query by eliminating unsuccessful intermediate results. For this, we propose a matrix-based structure for monitoring data streams and estimate the number of final result tuples of the query and find out unsuccessful tuples by matrix multiplication operations. And then using these information, we process efficiently a multi-way join continuous query by filtering out the unsuccessful tuples in advance before actual evaluation of the query.

An embedded co-processor for accelerating window joins over uncertain data streams

In many new applications fields, such as surveillance, networks, and sensor technologies, the generating data are time-varying, unpredictable, uncertain, continuously arriving, and must be processed online. To keep up with these rapid data streams, accelerating data processing is one of the key problems in a UDSMS (Uncertain Data Stream Management System). Most of the algorithms that are proposed in the literature are implemented in software and can potentially result in CPU saturation. To greatly promote UDSMS processing speed, we focus on accelerating join processing, which is one of the most costly operations. We propose, for the first time, a hardware co-processor called UWJSP (Uncertain data Window Join Special co-Processor) for implementation. UWJSP is composed of several processing units. Each unit executes special instructions that continuously track changing queries. We also propose an integrated method for improving the scalability and flexibility. The experimental results reveal that UWJSP can achieve an order of magnitude improvement over a software implementation. The low cost and high performance UWJSP can be used in applications where real-time performance is indispensable.

SensorStream: a semantic real-time stream management system

As data proliferates at increasing rates, the need for real-time stream processing applications increases as well. In the same way that Data Stream Management Systems (DSMS) have emerged from the database community, there is now a similar concern in managing dynamic knowledge among the Semantic Web community. Unfortunately, early relevant approaches are, to a large extent, theoretical and do not present convincing evidence of their efficiency in real dynamic environments. In this paper, we present a framework for the effective, real-time processing of streaming data and we define and analyse in depth its key components. Our framework serves as a basis for the implementation of the SensorStream prototype, on which we run numerous performance and scalability measurements that outline its behaviour and demonstrate its suitability and scalability for solutions that require real-time information processing from distributed and heterogeneous data sources.

Data Stream Management System and Capital Market Surveillance

Importance of real-time data analysis has been felt since early '90s and thus processing of streaming data (from either sensor networks or telecom switches or web and other disparate systems) is the demand of the industries worldwide. Quicker detection of fraudulent activities in a financial system is the order of the day. Thus capital market surveillance, if can be performed by using the streaming input of various trading transactions, without being stored, that would be beneficial to the regulatory authorities and stock exchanges. In this paper, we describe how stream processing using a data stream management system (DSMS) can be used for the above task and how effective would be that in terms of performance and latency. We present results obtained from using a commercial event stream processing system (IBM InfoSphere Streams platform) for certain typical fraud detection scenarios.

실시간 헬스케어 시스템을 위한 데이터 스트림 서버의 설계 및 구현

The EMR management services can monitoring the patients` record with any doctors in any hospital by using the internet and smartphones online. To handle the real time, multidimensional, continuous data, database management systems (DBMS) must cope with high insert rates for updates, however the traditional DBMS suffers from processing these kinds of data due to its serious design bottlenecks. So the researchers put forward to Data Stream Management System (DSMS). In this paper we describe the real-time Data Stream Server for Continuity of Care Record (CCR) that including continuos query processor. This system is compiled with DSMS and DBMS in EMR system for processing and monitoring the coming CCR data stream, and also storing the processed result with high-efficiency. The system enables users not only to query stored CCR information from DBMS, but also to execute continue query on real-time CCR Data Stream, and health information can be transferred between different healthcare providers that would reduce medical error. At last, we develop a IPhone mobile application to test the proposed real-time data stream server.