Querying Data Streams Research Articles

Planning operators of concurrent RDF stream processing queries

RDF stream processing (RSP), which aims to query data streams and linked datasets using common data model and query languages extended from RDF and SPARQL, is gaining popularity. However, most of the existing RSP engines do not provide any optimisation techniques for shared join operators among query plans from concurrent queries. Many number of shared join operators can incur the waste of a lot of CPU resources like a processing memory. Moreover, queries on shared operators cause a slow response time because they must be re-evaluated without reusing intermediate results. To solve these problems, we propose an efficient method of optimising query plans on multiple queries. First, the proposed method evicts some data that get notified from the streams in order to maintain an efficient memory usage. Second, the proposed method optimises query plans to maximise the reuse of shared join results. Experimental results show that the proposed method has significant improvements in terms of memory consumption and latency, compared to the state-the-of-art methods.

Planning operators of concurrent RDF stream processing queries

RDF stream processing (RSP), which aims to query data streams and linked datasets using common data model and query languages extended from RDF and SPARQL, is gaining popularity. However, most of the existing RSP engines do not provide any optimisation techniques for shared join operators among query plans from concurrent queries. Many number of shared join operators can incur the waste of a lot of CPU resources like a processing memory. Moreover, queries on shared operators cause a slow response time because they must be re-evaluated without reusing intermediate results. To solve these problems, we propose an efficient method of optimising query plans on multiple queries. First, the proposed method evicts some data that get notified from the streams in order to maintain an efficient memory usage. Second, the proposed method optimises query plans to maximise the reuse of shared join results. Experimental results show that the proposed method has significant improvements in terms of memory consumption and latency, compared to the state-the-of-art methods.

Sliding windows over uncertain data streams

Uncertain data streams can have tuples with both value and existential uncertainty. A tuple has value uncertainty when it can assume multiple possible values. A tuple is existentially uncertain when the sum of the probabilities of its possible values is $$<$$ 1. A situation where existential uncertainty can arise is when applying relational operators to streams with value uncertainty. Several prior works have focused on querying and mining data streams with both value and existential uncertainty. However, none of them have studied, in depth, the implications of existential uncertainty on sliding window processing, even though it naturally arises when processing uncertain data. In this work, we study the challenges arising from existential uncertainty, more specifically the management of count-based sliding windows, which are a basic building block of stream processing applications. We extend the semantics of sliding window to define the novel concept of uncertain sliding windows and provide both exact and approximate algorithms for managing windows under existential uncertainty. We also show how current state-of-the-art techniques for answering similarity join queries can be easily adapted to be used with uncertain sliding windows. We evaluate our proposed techniques under a variety of configurations using real data. The results show that the algorithms used to maintain uncertain sliding windows can efficiently operate while providing a high-quality approximation in query answering. In addition, we show that sort-based similarity join algorithms can perform better than index-based techniques (on 17 real datasets) when the number of possible values per tuple is low, as in many real-world applications.

Comparing a knowledge-based and a data-driven method in querying data streams for system fault detection: A hydraulic drive system application

The field of fault detection and diagnosis has been the subject of considerable interest in industry. Fault detection may increase the availability of products, thereby improving their quality. Fault detection and diagnosis methods can be classified in three categories: data-driven, analytically based, and knowledge-based methods.In this work, we investigated the ability and the performance of applying two fault detection methods to query data streams produced from hydraulic drive systems. A knowledge-based method was compared to a data-driven method. A fault detection system based on a data stream management system (DSMS) was developed in order to test and compare the two methods using data from real hydraulic drive systems.The knowledge-based method was based on causal models (fault trees), and principal component analysis (PCA) was used to build the data-driven model. The performance of the methods in terms of accuracy and speed, was examined using normal and physically simulated fault data. The results show that both methods generate queries fast enough to query the data streams online, with a similar level of fault detection accuracy. The industrial applications of both methods include monitoring of individual industrial mechanical systems as well as fleets of such systems. One can conclude that both methods may be used to increase industrial system availability.

Préférences contextuelles pour la personnalisation des requêtes sur flux de données

Preference queries have been largely studied for relational systems but few propo- sals exist for stream data systems. Most of the existing proposals concern the skyline, top-k or top-k dominating queries, coupled with the sliding-window operator. However, user preferences queries on data streams may be more sophisticated than skyline or top-k and may involve more expressive operations on streams. This paper improves the existing work on data stream query- answering personalization by proposing a solution to express and handle contextual preferences together with a large variety of queries including one-shot and continuous queries. It adopts a more expressive preference model supporting context-based preferences, allowing to capture a wide range of situations. We propose algorithms to implement the new preference operators on stream data and validate their performance on a real-world dataset of stock market streams.

Fay

Fay is a flexible platform for the efficient collection, processing, and analysis of software execution traces. Fay provides dynamic tracing through use of runtime instrumentation and distributed aggregation within machines and across clusters. At the lowest level, Fay can be safely extended with new tracing primitives, including even untrusted, fully optimized machine code, and Fay can be applied to running user-mode or kernel-mode software without compromising system stability. At the highest level, Fay provides a unified, declarative means of specifying what events to trace, as well as the aggregation, processing, and analysis of those events. We have implemented the Fay tracing platform for Windows and integrated it with two powerful, expressive systems for distributed programming. Our implementation is easy to use, can be applied to unmodified production systems, and provides primitives that allow the overhead of tracing to be greatly reduced, compared to previous dynamic tracing platforms. To show the generality of Fay tracing, we reimplement, in experiments, a range of tracing strategies and several custom mechanisms from existing tracing frameworks. Fay shows that modern techniques for high-level querying and data-parallel processing of disagreggated data streams are well suited to comprehensive monitoring of software execution in distributed systems. Revisiting a lesson from the late 1960s [Deutsch and Grant 1971], Fay also demonstrates the efficiency and extensibility benefits of using safe, statically verified machine code as the basis for low-level execution tracing. Finally, Fay establishes that, by automatically deriving optimized query plans and code for safe extensions, the expressiveness and performance of high-level tracing queries can equal or even surpass that of specialized monitoring tools.

Prediction-based load shedding for burst data streams

Overload management has become very important in telecommunication networks, especially in the case of monitoring network elements that generate time-varying and burst data streams. Efficient overload management improves the quality of provided services, reduces revenue leakage, and minimizes network failure detection time. We argue that in order to improve the quality of bursty data stream processing, data stream management systems (DSMS) should be introduced into network management infrastructures and should monitor and optimize the execution of data stream queries through load-shedding techniques. We introduce a new mechanism for data prioritization and implement a new load-shedding framework, partially based on the popular TelegraphCQ DSMS. Our approach significantly improves the accuracy of data stream query results by using a prediction module to rank the values using preferential ranks. The prediction module tracks and registers all symptoms of overload, and either prevents overload before it occurs, or minimizes its effects. We present empirical evidence showing that our solution significantly improves data stream query accuracy. The effectiveness of the framework has been verified using both artificial and real data streams collected from network elements.

A session‐oriented approach in modeling hierarchies of streams

AbstractData stream systems became recently the focus of intense research activity. Sensor measurements, financial data, network packets and log files can all be seen as data streams. The ability to query data streams is of increasing importance and has been identified as a crucial element for modern organizations and agencies. This article identifies an interesting class of applications where stream sessions may be organized into a hierarchical fashion—i.e. sessions may consist of subsessions. For example, log streams from call centers belong to different call sessions and call sessions consist of services subsessions. We may want to monitor statistics and perform accounting at any level on this hierarchy, relative to any other (higher) level (e.g. monitoring the average service session per call vs the average service session for the entire system). We argue that data streams of this kind have rich procedural semantics—i.e. behaviour—and therefore a semantically rich model should be used: a session may be defined by opening and closing conditions, may have data and methods and may consist of subsessions. We propose a simple conceptual model based on the notion of ‘session’—similar to a class in an object‐oriented environment having lifetime semantics. Queries on top of this schema can be formulated via HSA (hierarchical stream aggregate) expressions. We give an algorithm describing how stream data ‘flow down’ session hierarchies and discuss potential evaluation and optimization techniques for HSAs. We finally present NESTREAM, a prototype implementation incorporating many of these concepts. Copyright © 2007 John Wiley &amp; Sons, Ltd.

Sliding Window Based Method for Processing Continuous J-A Queries on Data Streams

Sliding window join aggregation continuous queries (J-A queries for short) are often used in data stream applications. The intuitive method for processing these queries is to construct steaming operator tree and execute the tree in pipeline. The space cost of this method is O(α×β) , where α and β are the sizes of two sliding windows respectively. To reduce the requirement of memory, which is the most expensive resource in data stream query processing system, two novel sliding window J-A continuous query processing algorithms IC and TC are presented in this paper, whose space cost are both O(α+β). Theoretical analysis and experimental results show that

PSoup: a system for streaming queries over streaming data

Abstract.Recent work on querying data streams has focused on systems where newly arriving data is processed and continuously streamed to the user in real time. In many emerging applications, however, ad hoc queries and/or intermittent connectivity also require the processing of data that arrives prior to query submission or during a period of disconnection. For such applications, we have developed PSoup, a system that combines the processing of ad hoc and continuous queries by treating data and queries symmetrically, allowing new queries to be applied to old data and new data to be applied to old queries. PSoup also supports intermittent connectivity by separating the computation of query results from the delivery of those results. PSoup builds on adaptive query-processing techniques developed in the Telegraph project at UC Berkeley. In this paper, we describe PSoup and present experiments that demonstrate the effectiveness of our approach.

Guest editor introduction: special section on online analysis and querying of continuous data streams

IN a number of application domains, data arrives continuously in the form of a stream and needs to be processed in an online fashion. For example, in the network installations of large Telecom and Internet service providers, detailed usage information (e.g., Call Detail Records or CDRs, IP traffic statistics due to SNMP/RMON polling, etc.) from different parts of the network needs to be continuously collected and analyzed for interesting trends. Other applications that generate rapid, continuous, and large volumes of stream data include transactions in retail chains, ATM, and credit card operations in banks, weather measurements, sensor networks, etc. Further, for many mission-critical tasks such as fraud/anomaly detection in Telecom networks, it is important to be able to answer queries in real time and infer interesting patterns online. As a result, recent years have witnessed an increasing interest in designing single-pass algorithms for querying and mining data streams that examine each element in the stream only once. The large volumes of stream data, real-time response requirements of streaming applications, and architecture of modern computers impose two additional constraints on algorithms for querying streams: 1) The time for processing each stream element must be small, and 2) the amount of memory available to the query processor is limited. Thus, the challenge is to develop algorithms that can summarize data streams in a concise, but reasonably accurate, synopsis that can be stored in the allotted (small) amount of memory and can be used to provide approximate answers to user queries with some guarantees on the approximation error. Given the plethora of streaming applications and the nontrivial computational challenges they pose, the timing for a special issue on the topic could not have been better. This special issue of Transactions on Knowledge and Data Engineering presents five papers that propose novel synopses structures and fundamental algorithmic techniques for analyzing and querying continuous data streams. Of the five papers, four explore the space/accuracy trade off of stream processing algorithms for important problems like clustering and distinct value estimation, and one addresses issues related to the semantics of query operators on (infinite) streams. The first paper by Guha et al. is illustrative of a general class of streaming algorithms based on the principle of divideand-conquer. Conceptually, the algorithm proposed in the paper partitions the input stream into chunks and computes a succinct summary for each chunk. Then, in subsequent steps, it repeatedly combines chunk summaries from the previous step to compute new summaries until the final desired summary for the stream is obtained. Guha et al. show how this divide-and-conquer approach can be used to compute k centers for a stream, where each intermediate summary is a set of OðkÞ centers. The end result is a deterministic constant-factor approximation algorithm for clustering data streams. In the second paper, Cormode et al. exploit properties of p-stable distributions to estimate, with high probability, the number of distinct elements in a stream. Essentially, given a vector of random variables from a p-stable distribution, the Lp norm of a stream can be computed by summing the variables, after weighting each variable with the frequency of the corresponding stream element. Thus, choosing random variables from a p-stable distribution with a small p yields the number of distinct values in the stream. Wavelet transforms have been shown to be effective for approximating the frequency distribution of data. In their paper, Gilbert et al. present a randomized “sketch”-based method for estimating, in a streaming environment, the top few Wavelet coefficients with the highest energy. A key contribution of the paper is a special construction (based on second-order Reed-Muller codes) of the random variables used in sketching, so that Wavelet coefficients (and arbitrary range-sum queries) can be obtained very fast. Furthermore, the random sketch synopses considered in the paper can also be used to estimate join sizes, histograms, quantiles, and frequent elements in a stream. Interesting questions arise, especially for infinitely long data streams, when we consider the semantics of query operators like joins of two or more streams or group-by operators on a single stream. The paper by Tucker et al. refers to the first category of operators as unbounded stateful operators which are those that need to maintain state with no upper bound in its size and, so, may run out of memory. The latter class of operators belong to the category of blocking operators, which are those that need to read the entire input before emitting a single output and might never produce a result (if the stream is infinite). In order to address the abovementioned problems posed by unbounded stateful and blocking operators, Tucker et al. enhance the streaming IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 15, NO. 3, MAY/JUNE 2003 513

A relational approach to querying data streams

Data streams are long, relatively unstructured sequences of characters that contain information such as electronic mail or a tape backup of various documents and reports created in an office. A conceptual framework is presented, using relational algebra and relational databases, within which data streams may be queried. As information is extracted from the data streams, it is put into a relational database that may be queried in the usual manner. The database schema evolves as the user's knowledge of the content of the data stream changes. Operators are defined in terms of relational algebra that can be used to extract data from a specially defined relation that contains all or part of the data stream. This approach to querying data streams permits the integration of unstructured data with structured data. The operators defined extend the functionality of relational algebra in much the same way that the join does relative to the basic operators select, project, union, difference, and Cartesian product. >