Data Stream Systems Research Articles

Dynamic and coordinated software reconfiguration in distributed data stream systems

While many systems have to provide 24 × 7 services with no acceptable downtime, they have to be able to cope with changes in their execution environment and in the requirements that they must comply, in which data stream processing is one example of system that has to evolve during its execution. On one hand, dynamic reconfiguration (i.e., the capability of evolving on-the-fly) is a desirable feature. On the other hand, stream systems may suffer with the disruption and overhead caused by the reconfiguration. Due to these conflicting requirements, safe and non-disruptive reconfiguration is still an open problem. In this paper, we propose and validate a non-disruptive reconfiguration approach for distributed data stream systems that support stateful components and intermittent connections. We present experimental evidence that our mechanism supports safe distributed reconfiguration and has negligible impact on availability and performance.

Practical Identification of Dynamic Precedence Criteria to Produce Critical Results from Big Data Streams

Abstract During periods of high volume, big data stream applications may not have enough resources to process all incoming tuples. To maximize the production of the most critical results under such resource shortages, a recent solution, PR (short for Preferential Result), utilizes both static criteria (defined at compile-time) and dynamic criteria (identified online at run-time) to prioritize the processing of tuples throughout the query pipeline. Unfortunately, locating the optimal criteria placement (i.e., where in the query pipeline to evaluate each prioritization criteria) is extremely compute-intensive and runs in exponential time. This makes PR impractical for complex big data stream systems. Our proposed criteria selection and placement approach, PR-Prune (short for Preferential Result-Pruning), is practical. PR-Prune prunes ineffective dynamic criteria and combines multiple criteria along the same pipeline. To achieve this, PR-Prune seeks to expand the duration in the query pipeline that tuples identified as critical are pulled forward. Our experiments use a real data stream from the S&P 500 stocks, synthetic data streams, and a diverse set of queries. The results substantiate that PR-Prune increases the production of the most critical results compared to the state-of-the-art approaches. In addition, PR-Prune significantly lowers the optimization search time compared to PR.

Placement-Safe Operator-Graph Changes in Distributed Heterogeneous Data Stream Systems

Data stream processing systems enable querying continuous data without first storing it. Data stream queries may combine data from distributed data sources like different sensors in an environmental sensing application. This suggests distributed query processing. Thus the amount of transferred data can be reduced and more processing resources are available. However, distributed query processing on probably heterogeneous platforms complicates query optimization. This article investigates query optimization through operator graph changes and its interaction with operator placement on heterogeneous distributed systems. Pre-placement operator graph changes may prevent certain operator placements. Thereby the resource consumption of the query execution may unexpectedly increase. Based on the operator placement problem modeled as a task assignment problem (TAP), we prove that it is NP-hard to decide in general whether an arbitrary operator graph change may negatively influence the best possible TAP solution. We present conditions for several specific operator graph changes that guarantee to preserve the best possible TAP solution.

SURVEY OF STREAMING DATA WAREHOUSE UPDATE SCHEDULING

In this paper, we study scheduling problem of updates for the streaming data warehouses. The streaming data warehouses are the combination of traditional data warehouses and data stream systems. In this, jobs are nothing but the processes which are responsible for loading new data in the tables. Its purpose is to decrease the data staleness. In addition, it handles well, the challenges faced by the streaming warehouses like, data consistency, view hierarchies, heterogeneity found in update jobs because of dissimilar arrival times as well as size of data, preempt updates etc. The staleness of data is the scheduling metric considered here. In this, jobs are nothing but the processes which are responsible for loading new data in the tables. Its purpose is to decrease the data staleness. In addition, it handles well, the challenges faced by the streaming warehouses like, data consistency, view hierarchies, heterogeneity found in update jobs because of dissimilar arrival times as well as size of data, preempt updates etc. The staleness of data is the scheduling metric considered here.

Development of a Mesoscale Ensemble Data Assimilation System at the Naval Research Laboratory

Abstract An ensemble Kalman filter (EnKF) has been adopted and implemented at the Naval Research Laboratory (NRL) for mesoscale and storm-scale data assimilation to study the impact of ensemble assimilation of high-resolution observations, including those from Doppler radars, on storm prediction. The system has been improved during its implementation at NRL to further enhance its capability of assimilating various types of meteorological data. A parallel algorithm was also developed to increase the system’s computational efficiency on multiprocessor computers. The EnKF has been integrated into the NRL mesoscale data assimilation system and extensively tested to ensure that the system works appropriately with new observational data stream and forecast systems. An innovative procedure was developed to evaluate the impact of assimilated observations on ensemble analyses with no need to exclude any observations for independent validation (as required by the conventional evaluation based on data-denying experiments). The procedure was employed in this study to examine the impacts of ensemble size and localization on data assimilation and the results reveal a very interesting relationship between the ensemble size and the localization length scale. All the tests conducted in this study demonstrate the capabilities of the EnKF as a research tool for mesoscale and storm-scale data assimilation with potential operational applications.

Scalable Scheduling of Updates in Streaming Data Warehouses

We discuss update scheduling in streaming data warehouses, which combine the features of traditional data warehouses and data stream systems. In our setting, external sources push append-only data streams into the warehouse with a wide range of interarrival times. While traditional data warehouses are typically refreshed during downtimes, streaming warehouses are updated as new data arrive. We model the streaming warehouse update problem as a scheduling problem, where jobs correspond to processes that load new data into tables, and whose objective is to minimize data staleness over time (at time t, if a table has been updated with information up to some earlier time r, its staleness is t minus r). We then propose a scheduling framework that handles the complications encountered by a stream warehouse: view hierarchies and priorities, data consistency, inability to preempt updates, heterogeneity of update jobs caused by different interarrival times and data volumes among different sources, and transient overload. A novel feature of our framework is that scheduling decisions do not depend on properties of update jobs (such as deadlines), but rather on the effect of update jobs on data staleness. Finally, we present a suite of update scheduling algorithms and extensive simulation experiments to map out factors which affect their performance.

Increasing Robustness of RC4 Family for Automated Selection of Ciphersuites

Information security is dependent on various access control mechanism governed by cryptography or the art of encryption and decryption. Cryptography is the largely built in computer hardware or in software using various discrete structure. Security is, thus, merging in network with cryptography to provide secure communication between trusted and/or untrusted network. Efficient mechanism of encryption process is a primary method of protecting valuable electronic information. The encryption process also needs to be dynamic in order to face new hazards and advance methods used by cryptanalysts. RC4 is one of the most popular and efficient stream ciphers [1]. Stream ciphers are often used in applications where high speed and low delay are a requirement. This paper proposes to identify the security requirements for data stream systems; according to the increasing order of robustness. Various security circumstances demand various degree of security robustness. This paper suggests to develop a family of cryptographic algorithm based on the RC4 and checks on the performance of each one to analyze the robustness so that the particular algorithm becomes readily applicable to a circumstance. The applicability of algorithm is totally governed by the requirement of robustness for the security concern of the circumstance. The security concern of robustness of circumstances is matched against the designed ciphersuites where such family of algorithms is available. The process of selection of ciphersuites and hence the cryptographic algorithm is automated in order to ensure appropriate circumstantial robustness. This analysis shows that, the full-size RC4 remains secure against known attacks [2]. The family of algorithm considered here is based on RC4. The basic RC4 algorithm and its variants suggested by different authors like RC4 KSA [3], RC4-PRGA are included in the family. In order enhance the degree of security robustness, two new algorithm S-RC41 and S-RC42 are proposed. Addition of variants of RC4 increases the range of automatic selection ensuring further enhancement in security. RC4 family is analyzed for encryption, decryption and algorithm strength is analyzed. Robustness is determined considering both the factors; speed of encryption/decryption or performance and algorithm strength. It is proposed to evaluate the effect of existing RC4 changing different parameters.

Optimizing probabilistic query processing on continuous uncertain data

Uncertain data management is becoming increasingly important in many applications, in particular, in scientific databases and data stream systems. Uncertain data in these new environments is naturally modeled by continuous random variables. An important class of queries uses complex selection and join predicates and requires query answers to be returned if their existence probabilities pass a threshold. In this work, we optimize threshold query processing for continuous uncertain data by ( i ) expediting joins using new indexes on uncertain data, ( ii ) expediting selections by reducing dimensionality of integration and using faster filters, and ( iii ) optimizing a query plan using a dynamic, per-tuple based approach. Evaluation results using real-world data and benchmark queries show the accuracy and efficiency of our techniques and significant performance gains over a state-of-the-art threshold query optimizer.

A Distributed Stream Query Optimization Framework through Integrated Planning and Deployment

This paper addresses the problem of optimizing multiple distributed stream queries that are executing simultaneously in distributed data stream systems. We argue that the static query optimization approach of "plan, then deployment" is inadequate for handling distributed queries involving multiple streams and node dynamics faced in distributed data stream systems and applications. Thus, the selection of an optimal execution plan in such dynamic and networked computing systems must consider operator ordering, reuse, network placement, and search space reduction. We propose to use hierarchical network partitions to exploit various opportunities for operator-level reuse while utilizing network characteristics to maintain a manageable search space during query planning and deployment. We develop top-down, bottom-up, and hybrid algorithms for exploiting operator-level reuse through hierarchical network partitions. Formal analysis is presented to establish the bounds on the search space and suboptimality of our algorithms. We have implemented our algorithms in the IFLOW system, an adaptive distributed stream management system. Through simulations and experiments using a prototype deployed on Emulab, we demonstrate the effectiveness of our framework and our algorithms.

Multi-query join algorithm based on sharing for XML publish/subscribe data stream systems

In XML publish/subscribe system,, multi-query join involves the processing of subscriptions of several XML documents, including XPath evaluation, comparison computation and time management. XML join algorithm based on sharing observes that much join computation can be shared by several subscriptions to save the computation time of join. Experimental results demonstrate that the algorithm is efficient and suitable for the system of millions of subscriptions.

A Cost-Based Approach to Adaptive Resource Management in Data Stream Systems

Data stream management systems need to adaptively control their resources, since stream characteristics and query workload may vary over time. In this paper, we investigate an approach to adaptive resource management for continuous sliding-window queries that adjusts window sizes and time granularities to keep resource usage within bounds. These two novel techniques differ from standard load shedding approaches based on sampling, as they ensure exact query answers for given user-defined quality of service specifications, even under query reoptimization. In order to quantify the effects of both techniques on the various operations in a query plan, we develop an appropriate cost model for estimating operator resource allocation in terms of memory usage and processing costs. A thorough experimental study not only validates the accuracy of our cost model but also demonstrates the efficacy and scalability of the proposed techniques.

Rate-Sensitive Load Shedding in Data Stream Systems

Traditional load shedding algorithms for data stream systems calculate current operator selectivity over several run periods and use them to determine where to shed load during the next run period. In this paper, we show that the current selectivity may change due to the implementation of load shedding. Our algorithm, called RLS, determines the optimum drop location by these changed selectivity rather than those pre-calculated values. Simulation results demonstrate that RLS achieves higher accuracy than traditional algorithms.

Operator scheduling in data stream systems

In many applications involving continuous data streams, data arrival is bursty and data rate fluctuates over time. Systems that seek to give rapid or real-time query responses in such an environment must be prepared to deal gracefully with bursts in data arrival without compromising system performance. We discuss one strategy for processing bursty streams - adaptive, load-aware scheduling of query operators to minimize resource consumption during times of peak load. We show that the choice of an operator scheduling strategy can have significant impact on the runtime system memory usage as well as output latency. Our aim is to design a scheduling strategy that minimizes the maximum runtime system memory while maintaining the output latency within prespecified bounds. We first present Chain scheduling, an operator scheduling strategy for data stream systems that is near-optimal in minimizing runtime memory usage for any collection of single-stream queries involving selections, projections, and foreign-key joins with stored relations. Chain scheduling also performs well for queries with sliding-window joins over multiple streams and multiple queries of the above types. However, during bursts in input streams, when there is a buildup of unprocessed tuples, Chain scheduling may lead to high output latency. We study the online problem of minimizing maximum runtime memory, subject to a constraint on maximum latency. We present preliminary observations, negative results, and heuristics for this problem. A thorough experimental evaluation is provided where we demonstrate the potential benefits of Chain scheduling and its different variants, compare it with competing scheduling strategies, and validate our analytical conclusions.

SQL Extensions to Enable Decision Support within Database Systems

Modern organizations store massive amount of information within their data warehouses to carry out complex analysis and decision support, such as identifying interesting trends, making unusual patterns to stand out and verifying hypotheses. Besides traditional data warehousing applications, a new class of data-intensive systems is emerging: applications in which the data is modeled not as persistent relations but rather as transient data streams. In this article we approach decision support from the database viewpoint. We briefly mention SQL features – the standard query language of database systems – and review several proposed extensions to make it a decision support-oriented language. Special emphasis is given to EMF SQL, an approach that has proven very useful in complex querying over traditional data ware-housing applications. We argue that this extension can also be useful in the context of stream data. Finally, we review and discuss decision support challenges over data stream systems.

Decision support queries on a tape-resident data warehouse

Data warehouses collect masses of operational data, allowing analysts to extract information by issuing decision support queries on the otherwise discarded data. In many application areas (e.g. telecommunications), the warehoused data sets are multiple terabytes in size. Parts of these data sets are stored on very large disk arrays, while the remainder is stored on tape-based tertiary storage (which is one to two orders of magnitude less expensive than on-line storage). However, the inherently sequential nature of access to tape-based tertiary storage makes the efficient access to tape-resident data difficult to accomplish through conventional databases. In this paper, we present a way to make access to a massive tape-resident data warehouse easy and efficient. Ad hoc decision support queries usually involve large scale and complex aggregation over the detail data. These queries are difficult to express in SQL, and frequently require self-joins on the detail data (which are prohibitively expensive on the disk-resident data and infeasible to compute on tape-resident data), or unnecessary multiple passes through the detail data. An extension to SQL, the extended multi feature SQL (EMF SQL) expresses complex aggregation computations in a clear manner without using self-joins. The detail data in a data warehouse usually represents a record of past activities, and therefore is temporal. We show that complex queries involving sequences can be easily expressed in EMF SQL. An EMF SQL query can be optimized to minimize the number of passes through the detail data required to evaluate the query, in many cases to only one pass. We describe an efficient query evaluation algorithm along with a query optimization algorithm that minimizes the number of passes through the detail data, and which minimizes the amount of main memory required to evaluate the query. These algorithms are useful not only in the context of tape-resident data warehouses but also in data stream systems which require similar processing techniques.