Elements In Data Stream Research Articles

A Generic Framework for Finding Special Quadratic Elements in Data Streams

A Generic Framework for Finding Special Quadratic Elements in Data Streams

A Cloud-Based Parallel Space-Saving Algorithm for Big Networking Data

As the network continues to evolve, completely analyzing the traffic requires immeasurable resources. In situations of processing enormous streaming data, the most significant k items (Top-k) are more interesting, and some streaming algorithms are deployed due to relatively limited memory and also limited processing time per item. Space-saving is such one of the most popular algorithms for computation of frequent and Top-k elements in data streams. In this paper, this algorithm is implemented in the cloud for analyzing big networking data, and an empirical formula of the counter number is derived for efficiently maintaining Top-k items. Meanwhile, easily understandable proof manner is presented to prove the merging ability of Space-saving algorithm, and some experiments are conducted to affirm the effectiveness of the algorithm.

Reliable distributed data stream management in mobile environments

The proliferation of sensor technology, especially in the context of embedded systems, has brought forward novel types of applications that make use of streams of continuously generated sensor data. Many applications like telemonitoring in healthcare or roadside traffic monitoring and control particularly require data stream management (DSM) to be provided in a distributed, yet reliable way. This is even more important when DSM applications are deployed in a failure-prone distributed setting including resource-limited mobile devices, for instance in applications which aim at remotely monitoring mobile patients. In this paper, we introduce a model for distributed and reliable DSM. The contribution of this paper is threefold. First, in analogy to the SQL isolation levels, we define levels of reliability and describe necessary consistency constraints for distributed DSM that specify the tolerated loss, delay, or re-ordering of data stream elements, respectively. Second, we use this model to design and analyze an algorithm for reliable distributed DSM, namely efficient coordinated operator checkpointing (ECOC). We show that ECOC provides lossless and delay-limited reliable data stream management and thus can be used in critical application domains such as healthcare, where the loss of data stream elements cannot be tolerated. Third, we present detailed performance evaluations of the ECOC algorithm running on mobile, resource-limited devices. In particular, we can show that ECOC provides a high level of reliability while, at the same time, featuring good performance characteristics with moderate resource consumption.

Finding top-k elements in data streams

Identifying the most frequent elements in a data stream is a well known and difficult problem. Identifying the most frequent elements for each individual, especially in very large populations, is even harder. The use of fast and small memory footprint algorithms is paramount when the number of individuals is very large. In many situations such analysis needs to be performed and kept up to date in near real time. Fortunately, approximate answers are usually adequate when dealing with this problem. This paper presents a new and innovative algorithm that addresses this problem by merging the commonly used counter-based and sketch-based techniques for top-k identification. The algorithm provides the top-k list of elements, their frequency and an error estimate for each frequency value. It also provides strong guarantees on the error estimate, order of elements and inclusion of elements in the list depending on their real frequency. Additionally the algorithm provides stochastic bounds on the error and expected error estimates. Telecommunications customer’s behavior and voice call data is used to present concrete results obtained with this algorithm and to illustrate improvements over previously existing algorithms.

The frequent items problem, under polynomial decay, in the streaming model

We consider the problem of estimating the frequency count of data stream elements under polynomial decay functions. In these settings every element in the stream is assigned with a time-decreasing weight, using a non-increasing polynomial function. Decay functions are used in applications where older data is less significant, less interesting or even less reliable than recent data. Consider a data stream of N elements drawn from a universe U . We propose three poly-logarithmic algorithms for the problem. The first one, deterministic, uses O ( 1 ϵ 2 log N ( log log N + log U ) ) bits, where ϵ ∈ ( 0 , 1 ) is the approximation parameter. The second one, probabilistic, uses O ( 1 ϵ 2 log N δ log 1 ϵ ) bits or O ( 1 ϵ 2 log N δ log N ) bits, depending on the decay function parameter, where δ ∈ ( 0 , 1 ) is the probability of failure. The third one, deterministic in the stochastic model, uses O ( 1 ϵ log U ) bits or O ( 1 ϵ 2 log N ) bits, also depending on the decay parameter as will be described in this paper. This variant of the problem is important and has many applications. To our knowledge, it has never been studied before.

Comments on “an integrated efficient solution for computing frequent and top-k elements in data streams”

We investigate a well-known algorithm, Space-Saving [Metwally et al. 2006], which has been proven efficient and effective at mining frequent elements in data streams. We discovered an error in one of the theorems in Metwally et al. [2006]. Experiments are conducted to illustrate the error.

A physical operator algebra for prioritized elements in data streams

Data stream management systems are a natural choice to efficiently process continuous queries over high volume data streams, e.g., to monitor sensor data or transaction streams. An immediate reaction on detected critical or security relevant situations is essential for a secure and economic operation, as in our scenario of monitoring decentralized energy systems, which realize geographically distributed energy generation processes. Without further provisions existing processing approaches may lead to a delay of critical or security relevant messages in high load situations, e.g., caused by bursts.

Improved Approximate Detection of Duplicates for Data Streams Over Sliding Windows

Detecting duplicates in data streams is an important problem that has a wide range of applications. In general, precisely detecting duplicates in an unbounded data stream is not feasible in most streaming scenarios, and, on the other hand, the elements in data streams are always time sensitive. These make it particular significant approximately detecting duplicates among newly arrived elements of a data stream within a fixed time frame. In this paper, we present a novel data structure, Decaying Bloom Filter (DBF), as an extension of the Counting Bloom Filter, that effectively removes stale elements as new elements continuously arrive over sliding windows. On the DBF basis we present an efficient algorithm to approximately detect duplicates over sliding windows. Our algorithm may produce false positive errors, but not false negative errors as in many previous results. We analyze the time complexity and detection accuracy, and give a tight upper bound of false positive rate. For a given space G bits and sliding window size W, our algorithm has an amortized time complexity of O(√G/W). Both analytical and experimental results on synthetic data demonstrate that our algorithm is superior in both execution time and detection accuracy to the previous results.

Range‐Efficient Counting of Distinct Elements in a Massive Data Stream

Efficient one-pass estimation of $F_0$, the number of distinct elements in a data stream, is a fundamental problem arising in various contexts in databases and networking. We consider range-efficient estimation of $F_0$: estimation of the number of distinct elements in a data stream where each element of the stream is not just a single integer but an interval of integers. We present a randomized algorithm which yields an (e, d)-approximation of $F_0$, with the following time and space complexities ($n$ is the size of the universe of the items): (1) The amortized processing time per interval is $O(\log{\frac{1}{\delta}}\log \frac{n}{\epsilon})$. (2) The workspace used is $O(\frac{1}{\epsilon^2}\log{\frac{1}{\delta}}\log n)$ bits. Our algorithm improves upon a previous algorithm by Bar-Yossef, Kumar and Sivakumar [Proceedings of the $13$th ACM-SIAM Symposium on Discrete Algorithms (SODA), 2002, pp. 623-632], which requires $O(\frac{1}{\epsilon^5} \log{\frac{1}{\delta}}\log^5 n)$ processing time per item. This algorithm can also be used to compute the max-dominance norm of a stream of multiple signals and significantly improves upon the previous best time and space bounds by Cormode and Muthukrishnan [Proceedings of the $11$th European Symposium on Algorithms (ESA), Lecture Notes in Comput. Sci. 2938, Springer, Berlin, 2003, pp. 148-160]. This algorithm also provides an efficient solution to the distinct summation problem, which arises during data aggregation in sensor networks [Proceedings of the $2$nd International Conference on Embedded Networked Sensor Systems, ACM Press, New York, 2004, pp. 250-262, Proceedings of the $20$th International Conference on Data Engineering (ICDE), 2004, pp. 449-460].

An integrated efficient solution for computing frequent and top- k elements in data streams

We propose an approximate integrated approach for solving both problems of finding the most popular k elements, and finding frequent elements in a data stream coming from a large domain. Our solution is space efficient and reports both frequent and top- k elements with tight guarantees on errors. For general data distributions, our top- k algorithm returns k elements that have roughly the highest frequencies; and it uses limited space for calculating frequent elements. For realistic Zipfian data, the space requirement of the proposed algorithm for solving the exact frequent elements problem decreases dramatically with the parameter of the distribution; and for top- k queries, the analysis ensures that only the top- k elements, in the correct order, are reported. The experiments, using real and synthetic data sets, show space reductions with hardly any loss in accuracy. Having proved the effectiveness of the proposed approach through both analysis and experiments, we extend it to be able to answer continuous queries about frequent and top- k elements. Although the problems of incremental reporting of frequent and top- k elements are useful in many applications, to the best of our knowledge, no solution has been proposed.